[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260321 02:57:15.697667 10068 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.9.213.62:39551
I20260321 02:57:15.699571 10068 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260321 02:57:15.700930 10068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 02:57:15.720355 10075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 02:57:15.720353 10074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 02:57:15.722934 10077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 02:57:15.723387 10068 server_base.cc:1061] running on GCE node
I20260321 02:57:17.066094 10068 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 02:57:17.066370 10068 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 02:57:17.066527 10068 hybrid_clock.cc:648] HybridClock initialized: now 1774061837066511 us; error 0 us; skew 500 ppm
I20260321 02:57:17.067377 10068 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 02:57:17.073238 10068 webserver.cc:492] Webserver started at http://127.9.213.62:43611/ using document root <none> and password file <none>
I20260321 02:57:17.074158 10068 fs_manager.cc:362] Metadata directory not provided
I20260321 02:57:17.074380 10068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 02:57:17.074786 10068 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 02:57:17.081164 10068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1774061835490702-10068-0/minicluster-data/master-0-root/instance:
uuid: "7132a6305bb84eb3b9a755194521dfa5"
format_stamp: "Formatted at 2026-03-21 02:57:17 on dist-test-slave-mz2z"
I20260321 02:57:17.090677 10068 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260321 02:57:17.096772 10083 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 02:57:17.098309 10068 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260321 02:57:17.098878 10068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1774061835490702-10068-0/minicluster-data/master-0-root
uuid: "7132a6305bb84eb3b9a755194521dfa5"
format_stamp: "Formatted at 2026-03-21 02:57:17 on dist-test-slave-mz2z"
I20260321 02:57:17.099296 10068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1774061835490702-10068-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1774061835490702-10068-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1774061835490702-10068-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 02:57:17.169510 10068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 02:57:17.171301 10068 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260321 02:57:17.171881 10068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 02:57:17.255096 10068 rpc_server.cc:307] RPC server started. Bound to: 127.9.213.62:39551
I20260321 02:57:17.255342 10134 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.213.62:39551 every 8 connection(s)
I20260321 02:57:17.263796 10135 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:17.286790 10135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5: Bootstrap starting.
I20260321 02:57:17.293071 10135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:17.294986 10135 log.cc:826] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5: Log is configured to *not* fsync() on all Append() calls
I20260321 02:57:17.299932 10135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5: No bootstrap required, opened a new log
I20260321 02:57:17.323514 10135 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7132a6305bb84eb3b9a755194521dfa5" member_type: VOTER }
I20260321 02:57:17.324303 10135 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:17.324611 10135 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7132a6305bb84eb3b9a755194521dfa5, State: Initialized, Role: FOLLOWER
I20260321 02:57:17.325449 10135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7132a6305bb84eb3b9a755194521dfa5" member_type: VOTER }
I20260321 02:57:17.326014 10135 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 02:57:17.326370 10135 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 02:57:17.326742 10135 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:17.331018 10135 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7132a6305bb84eb3b9a755194521dfa5" member_type: VOTER }
I20260321 02:57:17.331882 10135 leader_election.cc:304] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7132a6305bb84eb3b9a755194521dfa5; no voters:
I20260321 02:57:17.333990 10135 leader_election.cc:290] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260321 02:57:17.334481 10138 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:17.337194 10138 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5 [term 1 LEADER]: Becoming Leader. State: Replica: 7132a6305bb84eb3b9a755194521dfa5, State: Running, Role: LEADER
I20260321 02:57:17.338196 10138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7132a6305bb84eb3b9a755194521dfa5" member_type: VOTER }
I20260321 02:57:17.339545 10135 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5 [sys.catalog]: configured and running, proceeding with master startup.
I20260321 02:57:17.349982 10139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7132a6305bb84eb3b9a755194521dfa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7132a6305bb84eb3b9a755194521dfa5" member_type: VOTER } }
I20260321 02:57:17.350418 10140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7132a6305bb84eb3b9a755194521dfa5. Latest consensus state: current_term: 1 leader_uuid: "7132a6305bb84eb3b9a755194521dfa5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7132a6305bb84eb3b9a755194521dfa5" member_type: VOTER } }
I20260321 02:57:17.351020 10139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5 [sys.catalog]: This master's current role is: LEADER
I20260321 02:57:17.351148 10140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5 [sys.catalog]: This master's current role is: LEADER
I20260321 02:57:17.354527 10147 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260321 02:57:17.366328 10147 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260321 02:57:17.369563 10068 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260321 02:57:17.385799 10147 catalog_manager.cc:1357] Generated new cluster ID: 57ea761beade4d07b5bc904263a8cf0c
I20260321 02:57:17.386165 10147 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260321 02:57:17.405090 10147 catalog_manager.cc:1380] Generated new certificate authority record
I20260321 02:57:17.407323 10147 catalog_manager.cc:1514] Loading token signing keys...
I20260321 02:57:17.440959 10147 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5: Generated new TSK 0
I20260321 02:57:17.442255 10147 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260321 02:57:17.501420 10068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 02:57:17.510983 10157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 02:57:17.513285 10158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 02:57:17.516757 10068 server_base.cc:1061] running on GCE node
W20260321 02:57:17.516825 10160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 02:57:17.518141 10068 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 02:57:17.518433 10068 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 02:57:17.518738 10068 hybrid_clock.cc:648] HybridClock initialized: now 1774061837518718 us; error 0 us; skew 500 ppm
I20260321 02:57:17.519805 10068 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 02:57:17.524712 10068 webserver.cc:492] Webserver started at http://127.9.213.1:41823/ using document root <none> and password file <none>
I20260321 02:57:17.525372 10068 fs_manager.cc:362] Metadata directory not provided
I20260321 02:57:17.525640 10068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 02:57:17.525938 10068 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 02:57:17.527117 10068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1774061835490702-10068-0/minicluster-data/ts-0-root/instance:
uuid: "36fe4867413f4998b0aa2d13625f5edd"
format_stamp: "Formatted at 2026-03-21 02:57:17 on dist-test-slave-mz2z"
I20260321 02:57:17.534109 10068 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20260321 02:57:17.539321 10165 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 02:57:17.540385 10068 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260321 02:57:17.540720 10068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1774061835490702-10068-0/minicluster-data/ts-0-root
uuid: "36fe4867413f4998b0aa2d13625f5edd"
format_stamp: "Formatted at 2026-03-21 02:57:17 on dist-test-slave-mz2z"
I20260321 02:57:17.541081 10068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1774061835490702-10068-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1774061835490702-10068-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1774061835490702-10068-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 02:57:17.566536 10068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 02:57:17.568428 10068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 02:57:17.572404 10068 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 02:57:17.578086 10068 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 02:57:17.578390 10068 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 02:57:17.578680 10068 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 02:57:17.578869 10068 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 02:57:17.711720 10068 rpc_server.cc:307] RPC server started. Bound to: 127.9.213.1:46771
I20260321 02:57:17.711870 10227 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.213.1:46771 every 8 connection(s)
I20260321 02:57:17.719514 10068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 02:57:17.732196 10232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 02:57:17.737970 10233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 02:57:17.748342 10068 server_base.cc:1061] running on GCE node
W20260321 02:57:17.752307 10235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 02:57:17.753739 10068 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 02:57:17.754112 10068 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 02:57:17.754346 10068 hybrid_clock.cc:648] HybridClock initialized: now 1774061837754328 us; error 0 us; skew 500 ppm
I20260321 02:57:17.755604 10068 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 02:57:17.757757 10228 heartbeater.cc:344] Connected to a master server at 127.9.213.62:39551
I20260321 02:57:17.758234 10228 heartbeater.cc:461] Registering TS with master...
I20260321 02:57:17.759410 10228 heartbeater.cc:507] Master 127.9.213.62:39551 requested a full tablet report, sending...
I20260321 02:57:17.759518 10068 webserver.cc:492] Webserver started at http://127.9.213.2:41549/ using document root <none> and password file <none>
I20260321 02:57:17.760840 10068 fs_manager.cc:362] Metadata directory not provided
I20260321 02:57:17.761554 10068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 02:57:17.761955 10068 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 02:57:17.762809 10100 ts_manager.cc:194] Registered new tserver with Master: 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771)
I20260321 02:57:17.763546 10068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1774061835490702-10068-0/minicluster-data/ts-1-root/instance:
uuid: "44e26258105a4f16b7eb73d92056a7b5"
format_stamp: "Formatted at 2026-03-21 02:57:17 on dist-test-slave-mz2z"
I20260321 02:57:17.765210 10100 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59688
I20260321 02:57:17.771369 10068 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260321 02:57:17.776979 10240 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 02:57:17.778658 10068 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260321 02:57:17.778995 10068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1774061835490702-10068-0/minicluster-data/ts-1-root
uuid: "44e26258105a4f16b7eb73d92056a7b5"
format_stamp: "Formatted at 2026-03-21 02:57:17 on dist-test-slave-mz2z"
I20260321 02:57:17.779281 10068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1774061835490702-10068-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1774061835490702-10068-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1774061835490702-10068-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 02:57:17.817456 10068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 02:57:17.818964 10068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 02:57:17.820569 10068 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 02:57:17.823451 10068 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 02:57:17.823704 10068 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 02:57:17.823989 10068 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 02:57:17.824190 10068 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 02:57:17.938151 10068 rpc_server.cc:307] RPC server started. Bound to: 127.9.213.2:42025
I20260321 02:57:17.938567 10302 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.213.2:42025 every 8 connection(s)
I20260321 02:57:17.946870 10068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 02:57:17.955108 10306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 02:57:17.956758 10307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 02:57:17.959270 10303 heartbeater.cc:344] Connected to a master server at 127.9.213.62:39551
I20260321 02:57:17.959793 10303 heartbeater.cc:461] Registering TS with master...
I20260321 02:57:17.961117 10303 heartbeater.cc:507] Master 127.9.213.62:39551 requested a full tablet report, sending...
I20260321 02:57:17.964784 10100 ts_manager.cc:194] Registered new tserver with Master: 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025)
I20260321 02:57:17.966645 10100 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59700
W20260321 02:57:17.967909 10309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 02:57:17.968046 10068 server_base.cc:1061] running on GCE node
I20260321 02:57:17.969584 10068 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 02:57:17.969847 10068 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 02:57:17.970089 10068 hybrid_clock.cc:648] HybridClock initialized: now 1774061837970064 us; error 0 us; skew 500 ppm
I20260321 02:57:17.970621 10068 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 02:57:17.973325 10068 webserver.cc:492] Webserver started at http://127.9.213.3:44203/ using document root <none> and password file <none>
I20260321 02:57:17.973839 10068 fs_manager.cc:362] Metadata directory not provided
I20260321 02:57:17.974071 10068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 02:57:17.974370 10068 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 02:57:17.975468 10068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1774061835490702-10068-0/minicluster-data/ts-2-root/instance:
uuid: "299a4a5db6eb447b86e68881e324911c"
format_stamp: "Formatted at 2026-03-21 02:57:17 on dist-test-slave-mz2z"
I20260321 02:57:17.980566 10068 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260321 02:57:17.984360 10314 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 02:57:17.985462 10068 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260321 02:57:17.985834 10068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1774061835490702-10068-0/minicluster-data/ts-2-root
uuid: "299a4a5db6eb447b86e68881e324911c"
format_stamp: "Formatted at 2026-03-21 02:57:17 on dist-test-slave-mz2z"
I20260321 02:57:17.986157 10068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1774061835490702-10068-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1774061835490702-10068-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1774061835490702-10068-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 02:57:18.008721 10068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 02:57:18.010380 10068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 02:57:18.012655 10068 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 02:57:18.015622 10068 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 02:57:18.015887 10068 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 02:57:18.016261 10068 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 02:57:18.016460 10068 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 02:57:18.114768 10068 rpc_server.cc:307] RPC server started. Bound to: 127.9.213.3:39003
I20260321 02:57:18.114872 10376 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.213.3:39003 every 8 connection(s)
I20260321 02:57:18.133728 10377 heartbeater.cc:344] Connected to a master server at 127.9.213.62:39551
I20260321 02:57:18.134208 10377 heartbeater.cc:461] Registering TS with master...
I20260321 02:57:18.135143 10377 heartbeater.cc:507] Master 127.9.213.62:39551 requested a full tablet report, sending...
I20260321 02:57:18.137296 10100 ts_manager.cc:194] Registered new tserver with Master: 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:18.137544 10068 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014315542s
I20260321 02:57:18.138737 10100 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59704
I20260321 02:57:18.204703 10100 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59710:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260321 02:57:18.208882 10100 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260321 02:57:18.323287 10342 tablet_service.cc:1511] Processing CreateTablet for tablet b16dbc74e5dc4c4eb7e938db12adcfe5 (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260321 02:57:18.325299 10342 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b16dbc74e5dc4c4eb7e938db12adcfe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.327795 10341 tablet_service.cc:1511] Processing CreateTablet for tablet 5067bb4162bd4927839474abf3f445f3 (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260321 02:57:18.329380 10340 tablet_service.cc:1511] Processing CreateTablet for tablet 7138ac120f3f4b0689010baea6f7cbc0 (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260321 02:57:18.330351 10339 tablet_service.cc:1511] Processing CreateTablet for tablet 0c038d9ceb9f4044b53376479ddce5ac (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260321 02:57:18.330942 10341 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5067bb4162bd4927839474abf3f445f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.332988 10340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7138ac120f3f4b0689010baea6f7cbc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.336407 10338 tablet_service.cc:1511] Processing CreateTablet for tablet 1fc59958b093424e913602a709b24065 (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260321 02:57:18.337703 10339 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c038d9ceb9f4044b53376479ddce5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.342870 10336 tablet_service.cc:1511] Processing CreateTablet for tablet 71c12424eea94c21a4aad2052dbac60e (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260321 02:57:18.348081 10334 tablet_service.cc:1511] Processing CreateTablet for tablet cb21c42d55ec496bb609b5ea5e6f844f (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260321 02:57:18.349973 10338 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc59958b093424e913602a709b24065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.340845 10337 tablet_service.cc:1511] Processing CreateTablet for tablet 02d438ffde12473b90a7549c34955b8e (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260321 02:57:18.353781 10268 tablet_service.cc:1511] Processing CreateTablet for tablet b16dbc74e5dc4c4eb7e938db12adcfe5 (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260321 02:57:18.354207 10267 tablet_service.cc:1511] Processing CreateTablet for tablet 5067bb4162bd4927839474abf3f445f3 (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
W20260321 02:57:18.365922 10148 auto_rebalancer.cc:247] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260321 02:57:18.363112 10265 tablet_service.cc:1511] Processing CreateTablet for tablet 0c038d9ceb9f4044b53376479ddce5ac (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260321 02:57:18.361224 10336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71c12424eea94c21a4aad2052dbac60e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.360191 10266 tablet_service.cc:1511] Processing CreateTablet for tablet 7138ac120f3f4b0689010baea6f7cbc0 (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260321 02:57:18.370705 10334 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb21c42d55ec496bb609b5ea5e6f844f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.371608 10191 tablet_service.cc:1511] Processing CreateTablet for tablet 7138ac120f3f4b0689010baea6f7cbc0 (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260321 02:57:18.372908 10191 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7138ac120f3f4b0689010baea6f7cbc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.371618 10337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02d438ffde12473b90a7549c34955b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.372956 10264 tablet_service.cc:1511] Processing CreateTablet for tablet 1fc59958b093424e913602a709b24065 (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260321 02:57:18.355360 10335 tablet_service.cc:1511] Processing CreateTablet for tablet f5b2778b93ef446c91564114305c7881 (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260321 02:57:18.374605 10193 tablet_service.cc:1511] Processing CreateTablet for tablet b16dbc74e5dc4c4eb7e938db12adcfe5 (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260321 02:57:18.376719 10268 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b16dbc74e5dc4c4eb7e938db12adcfe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.385362 10192 tablet_service.cc:1511] Processing CreateTablet for tablet 5067bb4162bd4927839474abf3f445f3 (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260321 02:57:18.395406 10265 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c038d9ceb9f4044b53376479ddce5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.395195 10335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5b2778b93ef446c91564114305c7881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.401177 10263 tablet_service.cc:1511] Processing CreateTablet for tablet 02d438ffde12473b90a7549c34955b8e (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260321 02:57:18.410449 10193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b16dbc74e5dc4c4eb7e938db12adcfe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.407696 10189 tablet_service.cc:1511] Processing CreateTablet for tablet 1fc59958b093424e913602a709b24065 (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260321 02:57:18.391881 10190 tablet_service.cc:1511] Processing CreateTablet for tablet 0c038d9ceb9f4044b53376479ddce5ac (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260321 02:57:18.418072 10263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02d438ffde12473b90a7549c34955b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.421367 10262 tablet_service.cc:1511] Processing CreateTablet for tablet 71c12424eea94c21a4aad2052dbac60e (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260321 02:57:18.428403 10192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5067bb4162bd4927839474abf3f445f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.436888 10260 tablet_service.cc:1511] Processing CreateTablet for tablet cb21c42d55ec496bb609b5ea5e6f844f (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260321 02:57:18.424937 10261 tablet_service.cc:1511] Processing CreateTablet for tablet f5b2778b93ef446c91564114305c7881 (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260321 02:57:18.444185 10186 tablet_service.cc:1511] Processing CreateTablet for tablet f5b2778b93ef446c91564114305c7881 (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260321 02:57:18.447530 10187 tablet_service.cc:1511] Processing CreateTablet for tablet 71c12424eea94c21a4aad2052dbac60e (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260321 02:57:18.431334 10188 tablet_service.cc:1511] Processing CreateTablet for tablet 02d438ffde12473b90a7549c34955b8e (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260321 02:57:18.459570 10185 tablet_service.cc:1511] Processing CreateTablet for tablet cb21c42d55ec496bb609b5ea5e6f844f (DEFAULT_TABLE table=test-workload [id=c7755c5dc0b24426bb140db03570ab7d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260321 02:57:18.458055 10267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5067bb4162bd4927839474abf3f445f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.462025 10262 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71c12424eea94c21a4aad2052dbac60e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.462853 10266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7138ac120f3f4b0689010baea6f7cbc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.463689 10261 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5b2778b93ef446c91564114305c7881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.466265 10260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb21c42d55ec496bb609b5ea5e6f844f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.468511 10185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb21c42d55ec496bb609b5ea5e6f844f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.469770 10190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c038d9ceb9f4044b53376479ddce5ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.480296 10186 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5b2778b93ef446c91564114305c7881. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.483705 10264 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc59958b093424e913602a709b24065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.492993 10188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02d438ffde12473b90a7549c34955b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.501994 10187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71c12424eea94c21a4aad2052dbac60e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.514485 10189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc59958b093424e913602a709b24065. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:18.601392 10395 tablet_bootstrap.cc:492] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c: Bootstrap starting.
I20260321 02:57:18.638453 10396 tablet_bootstrap.cc:492] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5: Bootstrap starting.
I20260321 02:57:18.668509 10396 tablet_bootstrap.cc:654] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:18.671181 10395 tablet_bootstrap.cc:654] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:18.675052 10397 tablet_bootstrap.cc:492] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd: Bootstrap starting.
I20260321 02:57:18.690336 10397 tablet_bootstrap.cc:654] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:18.690985 10396 tablet_bootstrap.cc:492] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5: No bootstrap required, opened a new log
I20260321 02:57:18.691349 10396 ts_tablet_manager.cc:1403] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.008s
I20260321 02:57:18.694303 10395 tablet_bootstrap.cc:492] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c: No bootstrap required, opened a new log
I20260321 02:57:18.694913 10395 ts_tablet_manager.cc:1403] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c: Time spent bootstrapping tablet: real 0.098s user 0.007s sys 0.007s
I20260321 02:57:18.694342 10396 raft_consensus.cc:359] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.695276 10396 raft_consensus.cc:385] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:18.695623 10396 raft_consensus.cc:740] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Initialized, Role: FOLLOWER
I20260321 02:57:18.696419 10396 consensus_queue.cc:260] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.698552 10395 raft_consensus.cc:359] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.699544 10395 raft_consensus.cc:385] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:18.699940 10395 raft_consensus.cc:740] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Initialized, Role: FOLLOWER
I20260321 02:57:18.701164 10395 consensus_queue.cc:260] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.702462 10397 tablet_bootstrap.cc:492] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd: No bootstrap required, opened a new log
I20260321 02:57:18.702942 10397 ts_tablet_manager.cc:1403] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd: Time spent bootstrapping tablet: real 0.030s user 0.016s sys 0.000s
I20260321 02:57:18.706349 10397 raft_consensus.cc:359] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.707110 10397 raft_consensus.cc:385] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:18.707451 10397 raft_consensus.cc:740] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Initialized, Role: FOLLOWER
I20260321 02:57:18.709122 10303 heartbeater.cc:499] Master 127.9.213.62:39551 was elected leader, sending a full tablet report...
I20260321 02:57:18.710242 10396 ts_tablet_manager.cc:1434] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5: Time spent starting tablet: real 0.019s user 0.009s sys 0.009s
I20260321 02:57:18.711701 10396 tablet_bootstrap.cc:492] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5: Bootstrap starting.
I20260321 02:57:18.720757 10396 tablet_bootstrap.cc:654] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:18.721323 10377 heartbeater.cc:499] Master 127.9.213.62:39551 was elected leader, sending a full tablet report...
I20260321 02:57:18.721863 10395 ts_tablet_manager.cc:1434] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c: Time spent starting tablet: real 0.027s user 0.012s sys 0.015s
I20260321 02:57:18.723172 10395 tablet_bootstrap.cc:492] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c: Bootstrap starting.
I20260321 02:57:18.730515 10395 tablet_bootstrap.cc:654] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:18.740353 10396 tablet_bootstrap.cc:492] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5: No bootstrap required, opened a new log
I20260321 02:57:18.708190 10397 consensus_queue.cc:260] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.741043 10396 ts_tablet_manager.cc:1403] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.015s
I20260321 02:57:18.743662 10395 tablet_bootstrap.cc:492] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c: No bootstrap required, opened a new log
I20260321 02:57:18.744205 10395 ts_tablet_manager.cc:1403] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.006s
I20260321 02:57:18.744649 10396 raft_consensus.cc:359] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.745414 10396 raft_consensus.cc:385] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:18.745721 10396 raft_consensus.cc:740] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Initialized, Role: FOLLOWER
I20260321 02:57:18.747023 10396 consensus_queue.cc:260] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.748126 10395 raft_consensus.cc:359] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } }
I20260321 02:57:18.749001 10395 raft_consensus.cc:385] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:18.749315 10395 raft_consensus.cc:740] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Initialized, Role: FOLLOWER
I20260321 02:57:18.750072 10396 ts_tablet_manager.cc:1434] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260321 02:57:18.750488 10395 consensus_queue.cc:260] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } }
I20260321 02:57:18.751154 10396 tablet_bootstrap.cc:492] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5: Bootstrap starting.
I20260321 02:57:18.752964 10395 ts_tablet_manager.cc:1434] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c: Time spent starting tablet: real 0.008s user 0.005s sys 0.002s
I20260321 02:57:18.753935 10395 tablet_bootstrap.cc:492] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c: Bootstrap starting.
I20260321 02:57:18.754369 10228 heartbeater.cc:499] Master 127.9.213.62:39551 was elected leader, sending a full tablet report...
I20260321 02:57:18.758414 10396 tablet_bootstrap.cc:654] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:18.760959 10395 tablet_bootstrap.cc:654] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:18.771931 10397 ts_tablet_manager.cc:1434] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd: Time spent starting tablet: real 0.069s user 0.005s sys 0.006s
I20260321 02:57:18.773165 10395 tablet_bootstrap.cc:492] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c: No bootstrap required, opened a new log
I20260321 02:57:18.776623 10395 ts_tablet_manager.cc:1403] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.011s
I20260321 02:57:18.777102 10397 tablet_bootstrap.cc:492] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd: Bootstrap starting.
I20260321 02:57:18.773166 10396 tablet_bootstrap.cc:492] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5: No bootstrap required, opened a new log
I20260321 02:57:18.779474 10395 raft_consensus.cc:359] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.780381 10395 raft_consensus.cc:385] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:18.780524 10396 ts_tablet_manager.cc:1403] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.008s
I20260321 02:57:18.780756 10395 raft_consensus.cc:740] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Initialized, Role: FOLLOWER
I20260321 02:57:18.781883 10395 consensus_queue.cc:260] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.785006 10396 raft_consensus.cc:359] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.793560 10396 raft_consensus.cc:385] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:18.793934 10396 raft_consensus.cc:740] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Initialized, Role: FOLLOWER
I20260321 02:57:18.794730 10396 consensus_queue.cc:260] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.791435 10397 tablet_bootstrap.cc:654] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:18.795611 10402 raft_consensus.cc:493] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:18.796175 10402 raft_consensus.cc:515] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.797240 10396 ts_tablet_manager.cc:1434] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5: Time spent starting tablet: real 0.016s user 0.004s sys 0.002s
I20260321 02:57:18.798424 10396 tablet_bootstrap.cc:492] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5: Bootstrap starting.
I20260321 02:57:18.792747 10395 ts_tablet_manager.cc:1434] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
I20260321 02:57:18.799722 10395 tablet_bootstrap.cc:492] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c: Bootstrap starting.
I20260321 02:57:18.802114 10402 leader_election.cc:290] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025)
I20260321 02:57:18.805869 10396 tablet_bootstrap.cc:654] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:18.806627 10395 tablet_bootstrap.cc:654] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:18.832168 10395 tablet_bootstrap.cc:492] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c: No bootstrap required, opened a new log
I20260321 02:57:18.832850 10395 ts_tablet_manager.cc:1403] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.008s
I20260321 02:57:18.833210 10396 tablet_bootstrap.cc:492] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5: No bootstrap required, opened a new log
I20260321 02:57:18.833659 10396 ts_tablet_manager.cc:1403] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.021s
I20260321 02:57:18.834712 10397 tablet_bootstrap.cc:492] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd: No bootstrap required, opened a new log
I20260321 02:57:18.835198 10397 ts_tablet_manager.cc:1403] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd: Time spent bootstrapping tablet: real 0.058s user 0.012s sys 0.012s
I20260321 02:57:18.836025 10395 raft_consensus.cc:359] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.836320 10396 raft_consensus.cc:359] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.836823 10395 raft_consensus.cc:385] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:18.837049 10396 raft_consensus.cc:385] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:18.837476 10396 raft_consensus.cc:740] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Initialized, Role: FOLLOWER
I20260321 02:57:18.837482 10395 raft_consensus.cc:740] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Initialized, Role: FOLLOWER
I20260321 02:57:18.838259 10396 consensus_queue.cc:260] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.838543 10395 consensus_queue.cc:260] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.840725 10396 ts_tablet_manager.cc:1434] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260321 02:57:18.846717 10396 tablet_bootstrap.cc:492] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5: Bootstrap starting.
I20260321 02:57:18.843336 10397 raft_consensus.cc:359] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.847065 10397 raft_consensus.cc:385] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:18.847394 10397 raft_consensus.cc:740] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Initialized, Role: FOLLOWER
I20260321 02:57:18.848119 10397 consensus_queue.cc:260] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.849951 10395 ts_tablet_manager.cc:1434] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c: Time spent starting tablet: real 0.017s user 0.005s sys 0.001s
I20260321 02:57:18.850670 10397 ts_tablet_manager.cc:1434] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260321 02:57:18.850926 10395 tablet_bootstrap.cc:492] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c: Bootstrap starting.
I20260321 02:57:18.847582 10278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7138ac120f3f4b0689010baea6f7cbc0" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e26258105a4f16b7eb73d92056a7b5" is_pre_election: true
I20260321 02:57:18.853121 10278 raft_consensus.cc:2468] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 0.
I20260321 02:57:18.854909 10316 leader_election.cc:304] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 44e26258105a4f16b7eb73d92056a7b5; no voters:
I20260321 02:57:18.846463 10203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7138ac120f3f4b0689010baea6f7cbc0" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fe4867413f4998b0aa2d13625f5edd" is_pre_election: true
I20260321 02:57:18.855827 10402 raft_consensus.cc:2804] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:57:18.856209 10402 raft_consensus.cc:493] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:57:18.856287 10203 raft_consensus.cc:2468] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 0.
I20260321 02:57:18.856595 10402 raft_consensus.cc:3060] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:18.856684 10397 tablet_bootstrap.cc:492] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd: Bootstrap starting.
I20260321 02:57:18.864158 10395 tablet_bootstrap.cc:654] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:18.865870 10402 raft_consensus.cc:515] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.869309 10396 tablet_bootstrap.cc:654] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:18.867861 10402 leader_election.cc:290] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 election: Requested vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025)
I20260321 02:57:18.870227 10203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7138ac120f3f4b0689010baea6f7cbc0" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:18.871337 10203 raft_consensus.cc:3060] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:18.872704 10278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7138ac120f3f4b0689010baea6f7cbc0" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
W20260321 02:57:18.874796 10378 tablet.cc:2379] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260321 02:57:18.879130 10397 tablet_bootstrap.cc:654] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:18.885316 10278 raft_consensus.cc:3060] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:18.898049 10395 tablet_bootstrap.cc:492] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c: No bootstrap required, opened a new log
I20260321 02:57:18.898541 10395 ts_tablet_manager.cc:1403] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.010s
I20260321 02:57:18.902115 10395 raft_consensus.cc:359] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.903127 10395 raft_consensus.cc:385] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:18.903503 10395 raft_consensus.cc:740] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Initialized, Role: FOLLOWER
I20260321 02:57:18.904605 10395 consensus_queue.cc:260] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.908334 10395 ts_tablet_manager.cc:1434] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c: Time spent starting tablet: real 0.009s user 0.004s sys 0.008s
I20260321 02:57:18.910112 10395 tablet_bootstrap.cc:492] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c: Bootstrap starting.
I20260321 02:57:18.911207 10397 tablet_bootstrap.cc:492] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd: No bootstrap required, opened a new log
I20260321 02:57:18.911707 10397 ts_tablet_manager.cc:1403] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd: Time spent bootstrapping tablet: real 0.055s user 0.009s sys 0.007s
I20260321 02:57:18.914608 10397 raft_consensus.cc:359] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.915350 10397 raft_consensus.cc:385] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:18.915634 10397 raft_consensus.cc:740] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Initialized, Role: FOLLOWER
I20260321 02:57:18.916326 10397 consensus_queue.cc:260] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.918588 10397 ts_tablet_manager.cc:1434] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260321 02:57:18.919241 10395 tablet_bootstrap.cc:654] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:18.919516 10397 tablet_bootstrap.cc:492] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd: Bootstrap starting.
I20260321 02:57:18.924088 10278 raft_consensus.cc:2468] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 1.
I20260321 02:57:18.925482 10316 leader_election.cc:304] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 44e26258105a4f16b7eb73d92056a7b5; no voters:
I20260321 02:57:18.926338 10402 raft_consensus.cc:2804] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:18.927184 10203 raft_consensus.cc:2468] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 1.
I20260321 02:57:18.940578 10397 tablet_bootstrap.cc:654] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:18.944927 10396 tablet_bootstrap.cc:492] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5: No bootstrap required, opened a new log
I20260321 02:57:18.945878 10396 ts_tablet_manager.cc:1403] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5: Time spent bootstrapping tablet: real 0.099s user 0.036s sys 0.033s
I20260321 02:57:18.949214 10417 raft_consensus.cc:493] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:18.949759 10417 raft_consensus.cc:515] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.950045 10396 raft_consensus.cc:359] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } }
I20260321 02:57:18.950773 10396 raft_consensus.cc:385] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:18.951102 10396 raft_consensus.cc:740] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Initialized, Role: FOLLOWER
I20260321 02:57:18.952054 10402 raft_consensus.cc:697] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Becoming Leader. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Running, Role: LEADER
I20260321 02:57:18.952893 10203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fc59958b093424e913602a709b24065" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fe4867413f4998b0aa2d13625f5edd" is_pre_election: true
I20260321 02:57:18.952894 10402 consensus_queue.cc:237] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
W20260321 02:57:18.954792 10316 leader_election.cc:343] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:18.955510 10396 consensus_queue.cc:260] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } }
I20260321 02:57:18.958592 10396 ts_tablet_manager.cc:1434] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5: Time spent starting tablet: real 0.012s user 0.005s sys 0.002s
I20260321 02:57:18.959636 10396 tablet_bootstrap.cc:492] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5: Bootstrap starting.
I20260321 02:57:18.966183 10396 tablet_bootstrap.cc:654] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:18.967785 10417 leader_election.cc:290] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025)
I20260321 02:57:18.969238 10278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fc59958b093424e913602a709b24065" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e26258105a4f16b7eb73d92056a7b5" is_pre_election: true
I20260321 02:57:18.974220 10278 raft_consensus.cc:2468] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 0.
I20260321 02:57:18.976678 10401 raft_consensus.cc:493] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:18.977146 10316 leader_election.cc:304] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 44e26258105a4f16b7eb73d92056a7b5; no voters: 36fe4867413f4998b0aa2d13625f5edd
I20260321 02:57:18.977754 10401 raft_consensus.cc:515] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:18.979564 10417 raft_consensus.cc:2804] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:57:18.981778 10417 raft_consensus.cc:493] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:57:18.982210 10417 raft_consensus.cc:3060] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:18.985528 10397 tablet_bootstrap.cc:492] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd: No bootstrap required, opened a new log
I20260321 02:57:18.987836 10397 ts_tablet_manager.cc:1403] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd: Time spent bootstrapping tablet: real 0.069s user 0.018s sys 0.009s
I20260321 02:57:18.985973 10098 catalog_manager.cc:5671] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c reported cstate change: term changed from 0 to 1, leader changed from <none> to 299a4a5db6eb447b86e68881e324911c (127.9.213.3). New cstate: current_term: 1 leader_uuid: "299a4a5db6eb447b86e68881e324911c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: HEALTHY } } }
I20260321 02:57:18.988032 10395 tablet_bootstrap.cc:492] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c: No bootstrap required, opened a new log
I20260321 02:57:18.990048 10395 ts_tablet_manager.cc:1403] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c: Time spent bootstrapping tablet: real 0.080s user 0.024s sys 0.004s
I20260321 02:57:18.996480 10401 leader_election.cc:290] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:18.993265 10396 tablet_bootstrap.cc:492] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5: No bootstrap required, opened a new log
I20260321 02:57:18.998653 10396 ts_tablet_manager.cc:1403] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.009s
I20260321 02:57:18.997812 10397 raft_consensus.cc:359] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.001302 10397 raft_consensus.cc:385] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:19.001662 10397 raft_consensus.cc:740] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Initialized, Role: FOLLOWER
I20260321 02:57:19.002398 10397 consensus_queue.cc:260] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.010546 10402 raft_consensus.cc:493] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:19.010809 10417 raft_consensus.cc:515] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.011252 10402 raft_consensus.cc:515] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } }
I20260321 02:57:19.014868 10405 raft_consensus.cc:493] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:19.018752 10402 leader_election.cc:290] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025), 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771)
I20260321 02:57:19.015816 10405 raft_consensus.cc:515] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.018589 10203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fc59958b093424e913602a709b24065" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:19.018646 10417 leader_election.cc:290] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 election: Requested vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025)
I20260321 02:57:19.022507 10278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b16dbc74e5dc4c4eb7e938db12adcfe5" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e26258105a4f16b7eb73d92056a7b5" is_pre_election: true
I20260321 02:57:19.023609 10202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b16dbc74e5dc4c4eb7e938db12adcfe5" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fe4867413f4998b0aa2d13625f5edd" is_pre_election: true
I20260321 02:57:19.021286 10395 raft_consensus.cc:359] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.029361 10277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fc59958b093424e913602a709b24065" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
I20260321 02:57:19.031104 10395 raft_consensus.cc:385] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:19.031200 10277 raft_consensus.cc:3060] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.031500 10395 raft_consensus.cc:740] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Initialized, Role: FOLLOWER
W20260321 02:57:19.028838 10316 leader_election.cc:343] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:19.034871 10396 raft_consensus.cc:359] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.035835 10396 raft_consensus.cc:385] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:19.036202 10396 raft_consensus.cc:740] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Initialized, Role: FOLLOWER
I20260321 02:57:19.037150 10396 consensus_queue.cc:260] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.029491 10278 raft_consensus.cc:2468] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 0.
I20260321 02:57:19.048408 10405 leader_election.cc:290] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:19.036211 10395 consensus_queue.cc:260] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
W20260321 02:57:19.046196 10316 leader_election.cc:343] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:19.069273 10316 leader_election.cc:304] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 44e26258105a4f16b7eb73d92056a7b5; no voters: 36fe4867413f4998b0aa2d13625f5edd
I20260321 02:57:19.070713 10402 raft_consensus.cc:2804] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:57:19.071213 10402 raft_consensus.cc:493] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:57:19.071619 10402 raft_consensus.cc:3060] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.076165 10405 raft_consensus.cc:493] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:19.076792 10405 raft_consensus.cc:515] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.079389 10405 leader_election.cc:290] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:19.080240 10277 raft_consensus.cc:2468] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 1.
I20260321 02:57:19.083900 10396 ts_tablet_manager.cc:1434] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5: Time spent starting tablet: real 0.083s user 0.006s sys 0.002s
I20260321 02:57:19.076745 10395 ts_tablet_manager.cc:1434] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c: Time spent starting tablet: real 0.085s user 0.011s sys 0.000s
I20260321 02:57:19.085919 10395 tablet_bootstrap.cc:492] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c: Bootstrap starting.
I20260321 02:57:19.091046 10316 leader_election.cc:304] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 44e26258105a4f16b7eb73d92056a7b5; no voters: 36fe4867413f4998b0aa2d13625f5edd
I20260321 02:57:19.091549 10396 tablet_bootstrap.cc:492] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5: Bootstrap starting.
I20260321 02:57:19.101487 10417 raft_consensus.cc:2804] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:19.104286 10417 raft_consensus.cc:697] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Becoming Leader. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Running, Role: LEADER
I20260321 02:57:19.108539 10202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fc59958b093424e913602a709b24065" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fe4867413f4998b0aa2d13625f5edd" is_pre_election: true
I20260321 02:57:19.108179 10402 raft_consensus.cc:515] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } }
I20260321 02:57:19.108645 10417 consensus_queue.cc:237] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.127547 10397 ts_tablet_manager.cc:1434] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd: Time spent starting tablet: real 0.138s user 0.008s sys 0.016s
W20260321 02:57:19.127827 10242 leader_election.cc:343] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:19.132705 10277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b16dbc74e5dc4c4eb7e938db12adcfe5" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
I20260321 02:57:19.133495 10277 raft_consensus.cc:3060] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.134869 10395 tablet_bootstrap.cc:654] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:19.128911 10402 leader_election.cc:290] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 election: Requested vote from peers 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025), 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771)
I20260321 02:57:19.132141 10202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b16dbc74e5dc4c4eb7e938db12adcfe5" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:19.147675 10352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fc59958b093424e913602a709b24065" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299a4a5db6eb447b86e68881e324911c" is_pre_election: true
I20260321 02:57:19.148444 10397 tablet_bootstrap.cc:492] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd: Bootstrap starting.
I20260321 02:57:19.150588 10402 raft_consensus.cc:493] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:19.150759 10242 leader_election.cc:304] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44e26258105a4f16b7eb73d92056a7b5; no voters: 299a4a5db6eb447b86e68881e324911c, 36fe4867413f4998b0aa2d13625f5edd
I20260321 02:57:19.151121 10402 raft_consensus.cc:515] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.152482 10401 raft_consensus.cc:2749] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:57:19.153939 10402 leader_election.cc:290] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025)
I20260321 02:57:19.154685 10203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5067bb4162bd4927839474abf3f445f3" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fe4867413f4998b0aa2d13625f5edd" is_pre_election: true
I20260321 02:57:19.155151 10402 raft_consensus.cc:493] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:19.155593 10203 raft_consensus.cc:2468] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 0.
I20260321 02:57:19.155622 10402 raft_consensus.cc:515] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
W20260321 02:57:19.157599 10316 leader_election.cc:343] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:19.158337 10316 leader_election.cc:304] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 36fe4867413f4998b0aa2d13625f5edd; no voters:
I20260321 02:57:19.163092 10418 raft_consensus.cc:2804] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:57:19.164651 10418 raft_consensus.cc:493] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:57:19.165024 10418 raft_consensus.cc:3060] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.163733 10278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5067bb4162bd4927839474abf3f445f3" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e26258105a4f16b7eb73d92056a7b5" is_pre_election: true
I20260321 02:57:19.170481 10278 raft_consensus.cc:2468] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 0.
I20260321 02:57:19.176790 10402 leader_election.cc:290] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025)
I20260321 02:57:19.186349 10278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b2778b93ef446c91564114305c7881" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e26258105a4f16b7eb73d92056a7b5" is_pre_election: true
I20260321 02:57:19.183801 10202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b2778b93ef446c91564114305c7881" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fe4867413f4998b0aa2d13625f5edd" is_pre_election: true
W20260321 02:57:19.189869 10316 leader_election.cc:343] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:19.190696 10401 raft_consensus.cc:493] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:57:19.195235 10316 leader_election.cc:343] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:19.195937 10316 leader_election.cc:304] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c; no voters: 36fe4867413f4998b0aa2d13625f5edd, 44e26258105a4f16b7eb73d92056a7b5
I20260321 02:57:19.193678 10401 raft_consensus.cc:515] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.205217 10401 leader_election.cc:290] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:19.203523 10417 raft_consensus.cc:2749] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:57:19.209168 10352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb21c42d55ec496bb609b5ea5e6f844f" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299a4a5db6eb447b86e68881e324911c" is_pre_election: true
W20260321 02:57:19.210593 10242 leader_election.cc:343] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:19.209358 10277 raft_consensus.cc:2468] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 1.
I20260321 02:57:19.213291 10316 leader_election.cc:304] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 44e26258105a4f16b7eb73d92056a7b5; no voters: 36fe4867413f4998b0aa2d13625f5edd
I20260321 02:57:19.214295 10417 raft_consensus.cc:2804] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:19.215435 10098 catalog_manager.cc:5671] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c reported cstate change: term changed from 0 to 1, leader changed from <none> to 299a4a5db6eb447b86e68881e324911c (127.9.213.3). New cstate: current_term: 1 leader_uuid: "299a4a5db6eb447b86e68881e324911c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: HEALTHY } } }
I20260321 02:57:19.214864 10202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb21c42d55ec496bb609b5ea5e6f844f" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fe4867413f4998b0aa2d13625f5edd" is_pre_election: true
I20260321 02:57:19.220476 10202 raft_consensus.cc:2468] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 0.
I20260321 02:57:19.218724 10417 raft_consensus.cc:697] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Becoming Leader. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Running, Role: LEADER
I20260321 02:57:19.222698 10242 leader_election.cc:304] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36fe4867413f4998b0aa2d13625f5edd, 44e26258105a4f16b7eb73d92056a7b5; no voters: 299a4a5db6eb447b86e68881e324911c
I20260321 02:57:19.223501 10396 tablet_bootstrap.cc:654] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:19.225382 10401 raft_consensus.cc:2804] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:57:19.226504 10351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71c12424eea94c21a4aad2052dbac60e" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299a4a5db6eb447b86e68881e324911c" is_pre_election: true
I20260321 02:57:19.229048 10401 raft_consensus.cc:493] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:57:19.229108 10277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb21c42d55ec496bb609b5ea5e6f844f" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e26258105a4f16b7eb73d92056a7b5" is_pre_election: true
I20260321 02:57:19.226174 10352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb21c42d55ec496bb609b5ea5e6f844f" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299a4a5db6eb447b86e68881e324911c" is_pre_election: true
I20260321 02:57:19.225360 10417 consensus_queue.cc:237] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } }
I20260321 02:57:19.230355 10401 raft_consensus.cc:3060] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.232331 10278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71c12424eea94c21a4aad2052dbac60e" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e26258105a4f16b7eb73d92056a7b5" is_pre_election: true
W20260321 02:57:19.232899 10167 leader_election.cc:343] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003): Illegal state: must be running to vote when last-logged opid is not known
W20260321 02:57:19.240684 10167 leader_election.cc:343] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003): Illegal state: must be running to vote when last-logged opid is not known
W20260321 02:57:19.241500 10167 leader_election.cc:343] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:19.241901 10167 leader_election.cc:304] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36fe4867413f4998b0aa2d13625f5edd; no voters: 299a4a5db6eb447b86e68881e324911c, 44e26258105a4f16b7eb73d92056a7b5
I20260321 02:57:19.245575 10395 tablet_bootstrap.cc:492] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c: No bootstrap required, opened a new log
I20260321 02:57:19.245889 10426 raft_consensus.cc:2749] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:57:19.246624 10395 ts_tablet_manager.cc:1403] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c: Time spent bootstrapping tablet: real 0.161s user 0.014s sys 0.019s
I20260321 02:57:19.251355 10397 tablet_bootstrap.cc:654] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:19.258139 10418 raft_consensus.cc:515] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.260712 10395 raft_consensus.cc:359] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.261443 10395 raft_consensus.cc:385] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:19.261746 10395 raft_consensus.cc:740] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Initialized, Role: FOLLOWER
I20260321 02:57:19.264660 10401 raft_consensus.cc:515] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.266840 10277 raft_consensus.cc:2393] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 36fe4867413f4998b0aa2d13625f5edd in current term 1: Already voted for candidate 44e26258105a4f16b7eb73d92056a7b5 in this term.
I20260321 02:57:19.267835 10401 leader_election.cc:290] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 1 election: Requested vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:19.267319 10396 tablet_bootstrap.cc:492] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5: No bootstrap required, opened a new log
I20260321 02:57:19.269708 10202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5067bb4162bd4927839474abf3f445f3" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:19.270797 10396 ts_tablet_manager.cc:1403] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5: Time spent bootstrapping tablet: real 0.179s user 0.017s sys 0.010s
I20260321 02:57:19.271199 10203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb21c42d55ec496bb609b5ea5e6f844f" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:19.271322 10352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb21c42d55ec496bb609b5ea5e6f844f" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299a4a5db6eb447b86e68881e324911c"
I20260321 02:57:19.270964 10167 leader_election.cc:304] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 36fe4867413f4998b0aa2d13625f5edd; no voters: 299a4a5db6eb447b86e68881e324911c, 44e26258105a4f16b7eb73d92056a7b5
I20260321 02:57:19.272408 10202 raft_consensus.cc:3060] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.272300 10418 leader_election.cc:290] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 election: Requested vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025)
I20260321 02:57:19.274396 10203 raft_consensus.cc:3060] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.276484 10277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5067bb4162bd4927839474abf3f445f3" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
I20260321 02:57:19.277068 10277 raft_consensus.cc:3060] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.276984 10098 catalog_manager.cc:5671] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c reported cstate change: term changed from 0 to 1, leader changed from <none> to 299a4a5db6eb447b86e68881e324911c (127.9.213.3). New cstate: current_term: 1 leader_uuid: "299a4a5db6eb447b86e68881e324911c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:19.278663 10418 raft_consensus.cc:493] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:19.278613 10396 raft_consensus.cc:359] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.279467 10396 raft_consensus.cc:385] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:19.279213 10418 raft_consensus.cc:515] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.279764 10396 raft_consensus.cc:740] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Initialized, Role: FOLLOWER
I20260321 02:57:19.268642 10395 consensus_queue.cc:260] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.280957 10352 raft_consensus.cc:3060] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.280977 10396 consensus_queue.cc:260] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.285012 10397 tablet_bootstrap.cc:492] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd: No bootstrap required, opened a new log
I20260321 02:57:19.285564 10397 ts_tablet_manager.cc:1403] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd: Time spent bootstrapping tablet: real 0.154s user 0.018s sys 0.005s
I20260321 02:57:19.286053 10278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e26258105a4f16b7eb73d92056a7b5" is_pre_election: true
W20260321 02:57:19.289637 10316 leader_election.cc:343] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:19.289914 10396 ts_tablet_manager.cc:1434] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5: Time spent starting tablet: real 0.017s user 0.003s sys 0.003s
I20260321 02:57:19.289952 10277 raft_consensus.cc:2468] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 1.
I20260321 02:57:19.286435 10418 leader_election.cc:290] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025)
I20260321 02:57:19.287649 10201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fe4867413f4998b0aa2d13625f5edd" is_pre_election: true
W20260321 02:57:19.292791 10316 leader_election.cc:343] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:19.292881 10395 ts_tablet_manager.cc:1434] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c: Time spent starting tablet: real 0.044s user 0.005s sys 0.004s
I20260321 02:57:19.293285 10316 leader_election.cc:304] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c; no voters: 36fe4867413f4998b0aa2d13625f5edd, 44e26258105a4f16b7eb73d92056a7b5
I20260321 02:57:19.294807 10401 raft_consensus.cc:493] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:19.296829 10316 leader_election.cc:304] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 44e26258105a4f16b7eb73d92056a7b5; no voters:
I20260321 02:57:19.295872 10401 raft_consensus.cc:515] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.298507 10418 raft_consensus.cc:2749] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:57:19.299162 10401 leader_election.cc:290] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:19.300973 10351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02d438ffde12473b90a7549c34955b8e" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299a4a5db6eb447b86e68881e324911c" is_pre_election: true
I20260321 02:57:19.301420 10417 raft_consensus.cc:2804] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:19.301906 10417 raft_consensus.cc:697] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Becoming Leader. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Running, Role: LEADER
I20260321 02:57:19.302135 10397 raft_consensus.cc:359] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } }
W20260321 02:57:19.302577 10242 leader_election.cc:343] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:19.302845 10397 raft_consensus.cc:385] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:19.303231 10397 raft_consensus.cc:740] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Initialized, Role: FOLLOWER
I20260321 02:57:19.303578 10201 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02d438ffde12473b90a7549c34955b8e" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fe4867413f4998b0aa2d13625f5edd" is_pre_election: true
I20260321 02:57:19.304688 10395 tablet_bootstrap.cc:492] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c: Bootstrap starting.
I20260321 02:57:19.296401 10396 tablet_bootstrap.cc:492] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5: Bootstrap starting.
I20260321 02:57:19.302759 10417 consensus_queue.cc:237] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
W20260321 02:57:19.305248 10242 leader_election.cc:343] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:19.307422 10352 raft_consensus.cc:2468] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 1.
I20260321 02:57:19.307291 10242 leader_election.cc:304] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 44e26258105a4f16b7eb73d92056a7b5; no voters: 299a4a5db6eb447b86e68881e324911c, 36fe4867413f4998b0aa2d13625f5edd
I20260321 02:57:19.308692 10401 raft_consensus.cc:2749] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:57:19.310397 10397 consensus_queue.cc:260] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } }
I20260321 02:57:19.313704 10242 leader_election.cc:304] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 44e26258105a4f16b7eb73d92056a7b5; no voters:
I20260321 02:57:19.314642 10401 raft_consensus.cc:2804] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:19.317507 10203 raft_consensus.cc:2468] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 1.
I20260321 02:57:19.318665 10426 raft_consensus.cc:2749] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:57:19.322306 10401 raft_consensus.cc:697] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 1 LEADER]: Becoming Leader. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Running, Role: LEADER
I20260321 02:57:19.325444 10397 ts_tablet_manager.cc:1434] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd: Time spent starting tablet: real 0.039s user 0.006s sys 0.001s
I20260321 02:57:19.324344 10401 consensus_queue.cc:237] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.326551 10397 tablet_bootstrap.cc:492] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd: Bootstrap starting.
I20260321 02:57:19.328619 10396 tablet_bootstrap.cc:654] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:19.326419 10202 raft_consensus.cc:2468] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 1.
I20260321 02:57:19.327513 10395 tablet_bootstrap.cc:654] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:19.343632 10395 tablet_bootstrap.cc:492] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c: No bootstrap required, opened a new log
I20260321 02:57:19.344331 10395 ts_tablet_manager.cc:1403] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c: Time spent bootstrapping tablet: real 0.040s user 0.012s sys 0.007s
I20260321 02:57:19.348402 10395 raft_consensus.cc:359] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.349723 10395 raft_consensus.cc:385] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:19.352433 10395 raft_consensus.cc:740] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Initialized, Role: FOLLOWER
I20260321 02:57:19.352681 10397 tablet_bootstrap.cc:654] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:19.353904 10396 tablet_bootstrap.cc:492] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5: No bootstrap required, opened a new log
I20260321 02:57:19.354305 10396 ts_tablet_manager.cc:1403] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5: Time spent bootstrapping tablet: real 0.062s user 0.011s sys 0.009s
I20260321 02:57:19.353277 10395 consensus_queue.cc:260] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.357810 10395 ts_tablet_manager.cc:1434] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260321 02:57:19.358857 10395 tablet_bootstrap.cc:492] T 71c12424eea94c21a4aad2052dbac60e P 299a4a5db6eb447b86e68881e324911c: Bootstrap starting.
I20260321 02:57:19.358894 10396 raft_consensus.cc:359] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.359861 10396 raft_consensus.cc:385] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:19.360263 10396 raft_consensus.cc:740] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Initialized, Role: FOLLOWER
I20260321 02:57:19.361406 10396 consensus_queue.cc:260] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.364233 10396 ts_tablet_manager.cc:1434] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5: Time spent starting tablet: real 0.010s user 0.007s sys 0.002s
I20260321 02:57:19.371532 10097 catalog_manager.cc:5671] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c reported cstate change: term changed from 0 to 1, leader changed from <none> to 299a4a5db6eb447b86e68881e324911c (127.9.213.3). New cstate: current_term: 1 leader_uuid: "299a4a5db6eb447b86e68881e324911c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: HEALTHY } } }
W20260321 02:57:19.371544 10148 auto_rebalancer.cc:247] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260321 02:57:19.401315 10396 tablet_bootstrap.cc:492] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5: Bootstrap starting.
I20260321 02:57:19.402823 10397 tablet_bootstrap.cc:492] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd: No bootstrap required, opened a new log
I20260321 02:57:19.403432 10397 ts_tablet_manager.cc:1403] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd: Time spent bootstrapping tablet: real 0.077s user 0.011s sys 0.009s
I20260321 02:57:19.406738 10397 raft_consensus.cc:359] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.407497 10397 raft_consensus.cc:385] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:19.407802 10397 raft_consensus.cc:740] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Initialized, Role: FOLLOWER
I20260321 02:57:19.408720 10396 tablet_bootstrap.cc:654] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:19.408540 10397 consensus_queue.cc:260] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.409016 10098 catalog_manager.cc:5671] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2). New cstate: current_term: 1 leader_uuid: "44e26258105a4f16b7eb73d92056a7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:19.404018 10395 tablet_bootstrap.cc:654] T 71c12424eea94c21a4aad2052dbac60e P 299a4a5db6eb447b86e68881e324911c: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:19.414026 10397 ts_tablet_manager.cc:1434] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260321 02:57:19.415138 10397 tablet_bootstrap.cc:492] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd: Bootstrap starting.
I20260321 02:57:19.416092 10417 consensus_queue.cc:1048] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:57:19.450695 10397 tablet_bootstrap.cc:654] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:19.455919 10395 tablet_bootstrap.cc:492] T 71c12424eea94c21a4aad2052dbac60e P 299a4a5db6eb447b86e68881e324911c: No bootstrap required, opened a new log
I20260321 02:57:19.453135 10396 tablet_bootstrap.cc:492] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5: No bootstrap required, opened a new log
I20260321 02:57:19.456446 10395 ts_tablet_manager.cc:1403] T 71c12424eea94c21a4aad2052dbac60e P 299a4a5db6eb447b86e68881e324911c: Time spent bootstrapping tablet: real 0.098s user 0.008s sys 0.005s
I20260321 02:57:19.456732 10396 ts_tablet_manager.cc:1403] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5: Time spent bootstrapping tablet: real 0.056s user 0.017s sys 0.007s
I20260321 02:57:19.459553 10396 raft_consensus.cc:359] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.459404 10395 raft_consensus.cc:359] T 71c12424eea94c21a4aad2052dbac60e P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.460525 10396 raft_consensus.cc:385] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:19.460562 10395 raft_consensus.cc:385] T 71c12424eea94c21a4aad2052dbac60e P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:19.461045 10396 raft_consensus.cc:740] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Initialized, Role: FOLLOWER
I20260321 02:57:19.461220 10395 raft_consensus.cc:740] T 71c12424eea94c21a4aad2052dbac60e P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Initialized, Role: FOLLOWER
I20260321 02:57:19.461901 10396 consensus_queue.cc:260] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.462363 10395 consensus_queue.cc:260] T 71c12424eea94c21a4aad2052dbac60e P 299a4a5db6eb447b86e68881e324911c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.464877 10396 ts_tablet_manager.cc:1434] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260321 02:57:19.472723 10395 ts_tablet_manager.cc:1434] T 71c12424eea94c21a4aad2052dbac60e P 299a4a5db6eb447b86e68881e324911c: Time spent starting tablet: real 0.016s user 0.006s sys 0.003s
I20260321 02:57:19.487717 10397 tablet_bootstrap.cc:492] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd: No bootstrap required, opened a new log
I20260321 02:57:19.488567 10397 ts_tablet_manager.cc:1403] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd: Time spent bootstrapping tablet: real 0.074s user 0.012s sys 0.017s
I20260321 02:57:19.495389 10418 consensus_queue.cc:1048] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:19.492166 10397 raft_consensus.cc:359] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.498970 10397 raft_consensus.cc:385] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:19.499393 10397 raft_consensus.cc:740] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Initialized, Role: FOLLOWER
I20260321 02:57:19.500856 10397 consensus_queue.cc:260] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.503412 10397 ts_tablet_manager.cc:1434] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260321 02:57:19.505043 10397 tablet_bootstrap.cc:492] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd: Bootstrap starting.
I20260321 02:57:19.510360 10397 tablet_bootstrap.cc:654] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:19.523008 10397 tablet_bootstrap.cc:492] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd: No bootstrap required, opened a new log
I20260321 02:57:19.523510 10397 ts_tablet_manager.cc:1403] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.006s
I20260321 02:57:19.528569 10397 raft_consensus.cc:359] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.529309 10397 raft_consensus.cc:385] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:19.529603 10397 raft_consensus.cc:740] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Initialized, Role: FOLLOWER
I20260321 02:57:19.530253 10397 consensus_queue.cc:260] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.532786 10397 ts_tablet_manager.cc:1434] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260321 02:57:19.533849 10397 tablet_bootstrap.cc:492] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd: Bootstrap starting.
I20260321 02:57:19.540241 10397 tablet_bootstrap.cc:654] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:19.544085 10202 raft_consensus.cc:3060] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.550804 10418 consensus_queue.cc:1048] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:19.551005 10397 tablet_bootstrap.cc:492] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd: No bootstrap required, opened a new log
I20260321 02:57:19.551685 10397 ts_tablet_manager.cc:1403] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.013s
I20260321 02:57:19.554306 10397 raft_consensus.cc:359] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.554972 10397 raft_consensus.cc:385] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:19.555239 10397 raft_consensus.cc:740] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Initialized, Role: FOLLOWER
I20260321 02:57:19.556459 10397 consensus_queue.cc:260] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.558801 10397 ts_tablet_manager.cc:1434] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260321 02:57:19.567077 10417 consensus_queue.cc:1048] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:19.591203 10417 raft_consensus.cc:493] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:19.591718 10417 raft_consensus.cc:515] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.593304 10417 leader_election.cc:290] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025)
I20260321 02:57:19.594569 10202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02d438ffde12473b90a7549c34955b8e" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fe4867413f4998b0aa2d13625f5edd" is_pre_election: true
I20260321 02:57:19.594569 10277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02d438ffde12473b90a7549c34955b8e" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e26258105a4f16b7eb73d92056a7b5" is_pre_election: true
I20260321 02:57:19.595445 10277 raft_consensus.cc:2468] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 0.
I20260321 02:57:19.595423 10202 raft_consensus.cc:2468] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 0.
I20260321 02:57:19.596685 10316 leader_election.cc:304] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 36fe4867413f4998b0aa2d13625f5edd; no voters:
I20260321 02:57:19.597374 10417 raft_consensus.cc:2804] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:57:19.597735 10417 raft_consensus.cc:493] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:57:19.598076 10417 raft_consensus.cc:3060] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.603436 10417 raft_consensus.cc:515] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.605544 10417 leader_election.cc:290] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 election: Requested vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025)
I20260321 02:57:19.606442 10202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02d438ffde12473b90a7549c34955b8e" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:19.606573 10277 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02d438ffde12473b90a7549c34955b8e" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
I20260321 02:57:19.606976 10202 raft_consensus.cc:3060] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.607173 10277 raft_consensus.cc:3060] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.612139 10202 raft_consensus.cc:2468] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 1.
I20260321 02:57:19.613238 10316 leader_election.cc:304] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 36fe4867413f4998b0aa2d13625f5edd; no voters:
I20260321 02:57:19.613963 10417 raft_consensus.cc:2804] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:19.613988 10277 raft_consensus.cc:2468] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 1.
I20260321 02:57:19.615048 10417 raft_consensus.cc:697] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Becoming Leader. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Running, Role: LEADER
I20260321 02:57:19.615935 10417 consensus_queue.cc:237] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.625967 10096 catalog_manager.cc:5671] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c reported cstate change: term changed from 0 to 1, leader changed from <none> to 299a4a5db6eb447b86e68881e324911c (127.9.213.3). New cstate: current_term: 1 leader_uuid: "299a4a5db6eb447b86e68881e324911c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: HEALTHY } } }
I20260321 02:57:19.638697 10202 raft_consensus.cc:3060] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.648105 10417 consensus_queue.cc:1048] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:19.659464 10402 consensus_queue.cc:1048] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:19.659799 10426 raft_consensus.cc:493] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:19.660334 10426 raft_consensus.cc:515] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.662387 10426 leader_election.cc:290] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:19.665769 10352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b2778b93ef446c91564114305c7881" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299a4a5db6eb447b86e68881e324911c" is_pre_election: true
I20260321 02:57:19.666169 10276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b2778b93ef446c91564114305c7881" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e26258105a4f16b7eb73d92056a7b5" is_pre_election: true
I20260321 02:57:19.666538 10352 raft_consensus.cc:2468] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36fe4867413f4998b0aa2d13625f5edd in term 0.
I20260321 02:57:19.666851 10276 raft_consensus.cc:2468] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36fe4867413f4998b0aa2d13625f5edd in term 0.
I20260321 02:57:19.668488 10167 leader_election.cc:304] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36fe4867413f4998b0aa2d13625f5edd, 44e26258105a4f16b7eb73d92056a7b5; no voters:
I20260321 02:57:19.670801 10405 raft_consensus.cc:2804] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:57:19.671325 10405 raft_consensus.cc:493] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:57:19.671763 10405 raft_consensus.cc:3060] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.682147 10405 raft_consensus.cc:515] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.683995 10405 leader_election.cc:290] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 1 election: Requested vote from peers 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:19.685256 10352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b2778b93ef446c91564114305c7881" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299a4a5db6eb447b86e68881e324911c"
I20260321 02:57:19.685557 10278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b2778b93ef446c91564114305c7881" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
I20260321 02:57:19.685842 10352 raft_consensus.cc:3060] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.686283 10278 raft_consensus.cc:3060] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.693038 10278 raft_consensus.cc:2468] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fe4867413f4998b0aa2d13625f5edd in term 1.
I20260321 02:57:19.693038 10352 raft_consensus.cc:2468] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fe4867413f4998b0aa2d13625f5edd in term 1.
I20260321 02:57:19.694787 10167 leader_election.cc:304] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 36fe4867413f4998b0aa2d13625f5edd; no voters:
I20260321 02:57:19.695884 10405 raft_consensus.cc:2804] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:19.696523 10405 raft_consensus.cc:697] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [term 1 LEADER]: Becoming Leader. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Running, Role: LEADER
I20260321 02:57:19.697292 10405 consensus_queue.cc:237] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.707214 10098 catalog_manager.cc:5671] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd reported cstate change: term changed from 0 to 1, leader changed from <none> to 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1). New cstate: current_term: 1 leader_uuid: "36fe4867413f4998b0aa2d13625f5edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:19.735602 10401 raft_consensus.cc:493] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:19.736210 10401 raft_consensus.cc:515] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.738142 10401 leader_election.cc:290] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:19.739027 10202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71c12424eea94c21a4aad2052dbac60e" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fe4867413f4998b0aa2d13625f5edd" is_pre_election: true
I20260321 02:57:19.739423 10352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71c12424eea94c21a4aad2052dbac60e" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299a4a5db6eb447b86e68881e324911c" is_pre_election: true
I20260321 02:57:19.739703 10202 raft_consensus.cc:2468] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 0.
I20260321 02:57:19.740139 10352 raft_consensus.cc:2468] T 71c12424eea94c21a4aad2052dbac60e P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 0.
I20260321 02:57:19.740837 10242 leader_election.cc:304] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36fe4867413f4998b0aa2d13625f5edd, 44e26258105a4f16b7eb73d92056a7b5; no voters:
I20260321 02:57:19.741683 10401 raft_consensus.cc:2804] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:57:19.742079 10401 raft_consensus.cc:493] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:57:19.742429 10401 raft_consensus.cc:3060] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.750403 10401 raft_consensus.cc:515] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.752436 10401 leader_election.cc:290] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 1 election: Requested vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:19.753731 10352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71c12424eea94c21a4aad2052dbac60e" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299a4a5db6eb447b86e68881e324911c"
I20260321 02:57:19.753824 10202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "71c12424eea94c21a4aad2052dbac60e" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:19.754268 10352 raft_consensus.cc:3060] T 71c12424eea94c21a4aad2052dbac60e P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.754451 10202 raft_consensus.cc:3060] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.757539 10401 consensus_queue.cc:1048] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:19.761157 10352 raft_consensus.cc:2468] T 71c12424eea94c21a4aad2052dbac60e P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 1.
I20260321 02:57:19.762378 10242 leader_election.cc:304] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 44e26258105a4f16b7eb73d92056a7b5; no voters:
I20260321 02:57:19.762670 10202 raft_consensus.cc:2468] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 1.
I20260321 02:57:19.763439 10438 raft_consensus.cc:2804] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:19.775631 10438 raft_consensus.cc:697] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [term 1 LEADER]: Becoming Leader. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Running, Role: LEADER
I20260321 02:57:19.776618 10438 consensus_queue.cc:237] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.785377 10438 consensus_queue.cc:1048] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:19.787665 10096 catalog_manager.cc:5671] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2). New cstate: current_term: 1 leader_uuid: "44e26258105a4f16b7eb73d92056a7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:19.844720 10438 raft_consensus.cc:493] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:19.845232 10438 raft_consensus.cc:515] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.847103 10438 leader_election.cc:290] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:19.847918 10202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fe4867413f4998b0aa2d13625f5edd" is_pre_election: true
I20260321 02:57:19.848191 10351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299a4a5db6eb447b86e68881e324911c" is_pre_election: true
I20260321 02:57:19.848695 10202 raft_consensus.cc:2468] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 0.
I20260321 02:57:19.848893 10351 raft_consensus.cc:2468] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 0.
I20260321 02:57:19.849824 10242 leader_election.cc:304] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36fe4867413f4998b0aa2d13625f5edd, 44e26258105a4f16b7eb73d92056a7b5; no voters:
I20260321 02:57:19.850651 10438 raft_consensus.cc:2804] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:57:19.851326 10438 raft_consensus.cc:493] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:57:19.851723 10438 raft_consensus.cc:3060] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.857455 10438 raft_consensus.cc:515] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.859092 10438 leader_election.cc:290] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 1 election: Requested vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:19.860189 10202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:19.860296 10351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "299a4a5db6eb447b86e68881e324911c"
I20260321 02:57:19.861016 10202 raft_consensus.cc:3060] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.861003 10351 raft_consensus.cc:3060] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:19.863780 10418 consensus_queue.cc:1048] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:57:19.868680 10351 raft_consensus.cc:2468] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 1.
I20260321 02:57:19.868680 10202 raft_consensus.cc:2468] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 1.
I20260321 02:57:19.870008 10242 leader_election.cc:304] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 44e26258105a4f16b7eb73d92056a7b5; no voters:
I20260321 02:57:19.870769 10438 raft_consensus.cc:2804] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:19.871269 10438 raft_consensus.cc:697] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 1 LEADER]: Becoming Leader. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Running, Role: LEADER
I20260321 02:57:19.872112 10438 consensus_queue.cc:237] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:19.884052 10098 catalog_manager.cc:5671] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2). New cstate: current_term: 1 leader_uuid: "44e26258105a4f16b7eb73d92056a7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:19.891759 10457 consensus_queue.cc:1048] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:57:20.102846 10351 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b16dbc74e5dc4c4eb7e938db12adcfe5"
dest_uuid: "299a4a5db6eb447b86e68881e324911c"
mode: GRACEFUL
new_leader_uuid: "36fe4867413f4998b0aa2d13625f5edd"
from {username='slave'} at 127.0.0.1:34282
I20260321 02:57:20.103528 10351 raft_consensus.cc:606] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Received request to transfer leadership to TS 36fe4867413f4998b0aa2d13625f5edd
I20260321 02:57:20.109617 10351 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5067bb4162bd4927839474abf3f445f3"
dest_uuid: "299a4a5db6eb447b86e68881e324911c"
mode: GRACEFUL
new_leader_uuid: "36fe4867413f4998b0aa2d13625f5edd"
from {username='slave'} at 127.0.0.1:34282
I20260321 02:57:20.110356 10351 raft_consensus.cc:606] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Received request to transfer leadership to TS 36fe4867413f4998b0aa2d13625f5edd
I20260321 02:57:20.115861 10351 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7138ac120f3f4b0689010baea6f7cbc0"
dest_uuid: "299a4a5db6eb447b86e68881e324911c"
mode: GRACEFUL
new_leader_uuid: "36fe4867413f4998b0aa2d13625f5edd"
from {username='slave'} at 127.0.0.1:34282
I20260321 02:57:20.116568 10351 raft_consensus.cc:606] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Received request to transfer leadership to TS 36fe4867413f4998b0aa2d13625f5edd
I20260321 02:57:20.131546 10278 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac"
dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
mode: GRACEFUL
new_leader_uuid: "36fe4867413f4998b0aa2d13625f5edd"
from {username='slave'} at 127.0.0.1:35812
I20260321 02:57:20.132180 10278 raft_consensus.cc:606] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 1 LEADER]: Received request to transfer leadership to TS 36fe4867413f4998b0aa2d13625f5edd
I20260321 02:57:20.141443 10402 consensus_queue.cc:1048] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:20.143137 10351 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1fc59958b093424e913602a709b24065"
dest_uuid: "299a4a5db6eb447b86e68881e324911c"
mode: GRACEFUL
new_leader_uuid: "44e26258105a4f16b7eb73d92056a7b5"
from {username='slave'} at 127.0.0.1:34282
I20260321 02:57:20.148221 10351 raft_consensus.cc:606] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Received request to transfer leadership to TS 44e26258105a4f16b7eb73d92056a7b5
I20260321 02:57:20.162633 10417 consensus_queue.cc:1048] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:20.163228 10351 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "02d438ffde12473b90a7549c34955b8e"
dest_uuid: "299a4a5db6eb447b86e68881e324911c"
mode: GRACEFUL
new_leader_uuid: "44e26258105a4f16b7eb73d92056a7b5"
from {username='slave'} at 127.0.0.1:34282
I20260321 02:57:20.163816 10351 raft_consensus.cc:606] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Received request to transfer leadership to TS 44e26258105a4f16b7eb73d92056a7b5
I20260321 02:57:20.190846 10417 raft_consensus.cc:993] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c: : Instructing follower 44e26258105a4f16b7eb73d92056a7b5 to start an election
I20260321 02:57:20.191325 10417 raft_consensus.cc:1081] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Signalling peer 44e26258105a4f16b7eb73d92056a7b5 to start an election
I20260321 02:57:20.194617 10278 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "02d438ffde12473b90a7549c34955b8e"
dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
from {username='slave'} at 127.0.0.1:35800
I20260321 02:57:20.195254 10278 raft_consensus.cc:493] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:20.195642 10278 raft_consensus.cc:3060] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.204519 10405 consensus_queue.cc:1048] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:20.211266 10278 raft_consensus.cc:515] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.215355 10203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02d438ffde12473b90a7549c34955b8e" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:20.216027 10203 raft_consensus.cc:3060] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.223145 10203 raft_consensus.cc:2468] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 2.
I20260321 02:57:20.224644 10242 leader_election.cc:304] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36fe4867413f4998b0aa2d13625f5edd, 44e26258105a4f16b7eb73d92056a7b5; no voters:
I20260321 02:57:20.225543 10438 raft_consensus.cc:2804] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:57:20.226159 10438 raft_consensus.cc:697] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 2 LEADER]: Becoming Leader. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Running, Role: LEADER
I20260321 02:57:20.226756 10278 leader_election.cc:290] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 2 election: Requested vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:20.228648 10351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02d438ffde12473b90a7549c34955b8e" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "299a4a5db6eb447b86e68881e324911c"
I20260321 02:57:20.229295 10351 raft_consensus.cc:3055] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:57:20.229626 10351 raft_consensus.cc:740] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Running, Role: LEADER
I20260321 02:57:20.230456 10203 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f5b2778b93ef446c91564114305c7881"
dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
mode: GRACEFUL
new_leader_uuid: "44e26258105a4f16b7eb73d92056a7b5"
from {username='slave'} at 127.0.0.1:45154
I20260321 02:57:20.226960 10438 consensus_queue.cc:237] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.230409 10351 consensus_queue.cc:260] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.234781 10351 raft_consensus.cc:3060] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.240757 10203 raft_consensus.cc:606] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [term 1 LEADER]: Received request to transfer leadership to TS 44e26258105a4f16b7eb73d92056a7b5
I20260321 02:57:20.241474 10476 consensus_queue.cc:1048] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:20.242534 10099 catalog_manager.cc:5671] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 reported cstate change: term changed from 1 to 2, leader changed from 299a4a5db6eb447b86e68881e324911c (127.9.213.3) to 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2). New cstate: current_term: 2 leader_uuid: "44e26258105a4f16b7eb73d92056a7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:20.256520 10351 raft_consensus.cc:2468] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 2.
I20260321 02:57:20.257730 10405 raft_consensus.cc:993] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd: : Instructing follower 44e26258105a4f16b7eb73d92056a7b5 to start an election
I20260321 02:57:20.257592 10278 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cb21c42d55ec496bb609b5ea5e6f844f"
dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
mode: GRACEFUL
new_leader_uuid: "299a4a5db6eb447b86e68881e324911c"
from {username='slave'} at 127.0.0.1:35812
I20260321 02:57:20.258204 10278 raft_consensus.cc:606] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 1 LEADER]: Received request to transfer leadership to TS 299a4a5db6eb447b86e68881e324911c
I20260321 02:57:20.258149 10405 raft_consensus.cc:1081] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [term 1 LEADER]: Signalling peer 44e26258105a4f16b7eb73d92056a7b5 to start an election
I20260321 02:57:20.265340 10278 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f5b2778b93ef446c91564114305c7881"
dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
from {username='slave'} at 127.0.0.1:35802
I20260321 02:57:20.265930 10278 raft_consensus.cc:493] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:20.266304 10278 raft_consensus.cc:3060] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.274189 10278 raft_consensus.cc:515] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.277719 10203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b2778b93ef446c91564114305c7881" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:20.278386 10203 raft_consensus.cc:3055] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:57:20.278699 10278 leader_election.cc:290] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 2 election: Requested vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:20.278708 10203 raft_consensus.cc:740] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Running, Role: LEADER
I20260321 02:57:20.280547 10203 consensus_queue.cc:260] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.281687 10203 raft_consensus.cc:3060] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.279033 10351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5b2778b93ef446c91564114305c7881" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "299a4a5db6eb447b86e68881e324911c"
I20260321 02:57:20.290973 10351 raft_consensus.cc:3060] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.292349 10438 consensus_queue.cc:1048] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:57:20.301582 10203 raft_consensus.cc:2468] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 2.
I20260321 02:57:20.301867 10351 raft_consensus.cc:2468] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 2.
I20260321 02:57:20.303094 10242 leader_election.cc:304] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 44e26258105a4f16b7eb73d92056a7b5; no voters:
I20260321 02:57:20.304045 10438 raft_consensus.cc:2804] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:57:20.304553 10438 raft_consensus.cc:697] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 LEADER]: Becoming Leader. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Running, Role: LEADER
I20260321 02:57:20.305354 10438 consensus_queue.cc:237] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.314371 10099 catalog_manager.cc:5671] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5 reported cstate change: term changed from 1 to 2, leader changed from 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1) to 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2). New cstate: current_term: 2 leader_uuid: "44e26258105a4f16b7eb73d92056a7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:20.317530 10438 consensus_queue.cc:1048] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:20.446547 10480 consensus_queue.cc:1048] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:20.455415 10461 raft_consensus.cc:993] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5: : Instructing follower 36fe4867413f4998b0aa2d13625f5edd to start an election
I20260321 02:57:20.455924 10438 raft_consensus.cc:1081] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 1 LEADER]: Signalling peer 36fe4867413f4998b0aa2d13625f5edd to start an election
I20260321 02:57:20.459105 10203 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac"
dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
from {username='slave'} at 127.0.0.1:45148
I20260321 02:57:20.459842 10203 raft_consensus.cc:493] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:20.460305 10203 raft_consensus.cc:3060] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.462945 10438 consensus_queue.cc:1048] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:57:20.467859 10203 raft_consensus.cc:515] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.470141 10203 leader_election.cc:290] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 2 election: Requested vote from peers 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:20.471444 10202 raft_consensus.cc:1240] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Rejecting Update request from peer 44e26258105a4f16b7eb73d92056a7b5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:57:20.474014 10352 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "299a4a5db6eb447b86e68881e324911c"
I20260321 02:57:20.476284 10278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
I20260321 02:57:20.476897 10278 raft_consensus.cc:3055] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:57:20.477279 10278 raft_consensus.cc:740] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Running, Role: LEADER
I20260321 02:57:20.478021 10278 consensus_queue.cc:260] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.478927 10278 raft_consensus.cc:3060] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.483862 10402 raft_consensus.cc:993] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c: : Instructing follower 36fe4867413f4998b0aa2d13625f5edd to start an election
I20260321 02:57:20.484381 10457 raft_consensus.cc:1081] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Signalling peer 36fe4867413f4998b0aa2d13625f5edd to start an election
I20260321 02:57:20.486449 10202 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5067bb4162bd4927839474abf3f445f3"
dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
from {username='slave'} at 127.0.0.1:45140
I20260321 02:57:20.487218 10202 raft_consensus.cc:493] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:20.487819 10202 raft_consensus.cc:3060] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.491146 10278 raft_consensus.cc:2468] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fe4867413f4998b0aa2d13625f5edd in term 2.
I20260321 02:57:20.492430 10167 leader_election.cc:304] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 36fe4867413f4998b0aa2d13625f5edd, 44e26258105a4f16b7eb73d92056a7b5; no voters: 299a4a5db6eb447b86e68881e324911c
I20260321 02:57:20.493573 10405 raft_consensus.cc:2804] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:57:20.494074 10405 raft_consensus.cc:697] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 2 LEADER]: Becoming Leader. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Running, Role: LEADER
I20260321 02:57:20.494905 10405 consensus_queue.cc:237] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.495533 10202 raft_consensus.cc:515] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.497609 10202 leader_election.cc:290] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 2 election: Requested vote from peers 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:20.498767 10278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5067bb4162bd4927839474abf3f445f3" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
I20260321 02:57:20.498766 10351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5067bb4162bd4927839474abf3f445f3" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "299a4a5db6eb447b86e68881e324911c"
I20260321 02:57:20.499871 10351 raft_consensus.cc:3055] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:57:20.499754 10278 raft_consensus.cc:3060] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.500258 10351 raft_consensus.cc:740] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Running, Role: LEADER
I20260321 02:57:20.501405 10351 consensus_queue.cc:260] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.502547 10351 raft_consensus.cc:3060] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.507164 10096 catalog_manager.cc:5671] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd reported cstate change: term changed from 1 to 2, leader changed from 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2) to 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1). New cstate: current_term: 2 leader_uuid: "36fe4867413f4998b0aa2d13625f5edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:20.509827 10278 raft_consensus.cc:2468] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fe4867413f4998b0aa2d13625f5edd in term 2.
I20260321 02:57:20.511731 10167 leader_election.cc:304] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36fe4867413f4998b0aa2d13625f5edd, 44e26258105a4f16b7eb73d92056a7b5; no voters:
I20260321 02:57:20.512706 10405 raft_consensus.cc:2804] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:57:20.513180 10405 raft_consensus.cc:697] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 2 LEADER]: Becoming Leader. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Running, Role: LEADER
I20260321 02:57:20.514062 10405 consensus_queue.cc:237] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.520859 10351 raft_consensus.cc:2468] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fe4867413f4998b0aa2d13625f5edd in term 2.
I20260321 02:57:20.527647 10096 catalog_manager.cc:5671] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd reported cstate change: term changed from 1 to 2, leader changed from 299a4a5db6eb447b86e68881e324911c (127.9.213.3) to 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1). New cstate: current_term: 2 leader_uuid: "36fe4867413f4998b0aa2d13625f5edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:20.535784 10402 raft_consensus.cc:993] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c: : Instructing follower 36fe4867413f4998b0aa2d13625f5edd to start an election
I20260321 02:57:20.536316 10402 raft_consensus.cc:1081] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Signalling peer 36fe4867413f4998b0aa2d13625f5edd to start an election
I20260321 02:57:20.538100 10202 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7138ac120f3f4b0689010baea6f7cbc0"
dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
from {username='slave'} at 127.0.0.1:45140
I20260321 02:57:20.538736 10202 raft_consensus.cc:493] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:20.539125 10202 raft_consensus.cc:3060] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.545570 10202 raft_consensus.cc:515] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.547868 10202 leader_election.cc:290] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 2 election: Requested vote from peers 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:20.549582 10278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7138ac120f3f4b0689010baea6f7cbc0" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
I20260321 02:57:20.550355 10278 raft_consensus.cc:3060] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.550537 10351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7138ac120f3f4b0689010baea6f7cbc0" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "299a4a5db6eb447b86e68881e324911c"
I20260321 02:57:20.551347 10351 raft_consensus.cc:3055] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:57:20.551795 10351 raft_consensus.cc:740] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Running, Role: LEADER
I20260321 02:57:20.552852 10351 consensus_queue.cc:260] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.554036 10351 raft_consensus.cc:3060] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.559846 10351 raft_consensus.cc:2468] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fe4867413f4998b0aa2d13625f5edd in term 2.
I20260321 02:57:20.560207 10278 raft_consensus.cc:2468] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fe4867413f4998b0aa2d13625f5edd in term 2.
I20260321 02:57:20.561137 10167 leader_election.cc:304] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 36fe4867413f4998b0aa2d13625f5edd; no voters:
I20260321 02:57:20.561903 10405 raft_consensus.cc:2804] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:57:20.562373 10405 raft_consensus.cc:697] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 2 LEADER]: Becoming Leader. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Running, Role: LEADER
I20260321 02:57:20.563206 10405 consensus_queue.cc:237] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.572829 10096 catalog_manager.cc:5671] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd reported cstate change: term changed from 1 to 2, leader changed from 299a4a5db6eb447b86e68881e324911c (127.9.213.3) to 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1). New cstate: current_term: 2 leader_uuid: "36fe4867413f4998b0aa2d13625f5edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:20.614214 10461 raft_consensus.cc:993] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5: : Instructing follower 299a4a5db6eb447b86e68881e324911c to start an election
I20260321 02:57:20.614717 10438 raft_consensus.cc:1081] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 1 LEADER]: Signalling peer 299a4a5db6eb447b86e68881e324911c to start an election
I20260321 02:57:20.616392 10351 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cb21c42d55ec496bb609b5ea5e6f844f"
dest_uuid: "299a4a5db6eb447b86e68881e324911c"
from {username='slave'} at 127.0.0.1:34272
I20260321 02:57:20.616894 10351 raft_consensus.cc:493] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:20.617214 10351 raft_consensus.cc:3060] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.622133 10351 raft_consensus.cc:515] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.624002 10351 leader_election.cc:290] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 2 election: Requested vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025)
I20260321 02:57:20.625324 10202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb21c42d55ec496bb609b5ea5e6f844f" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:20.625191 10278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cb21c42d55ec496bb609b5ea5e6f844f" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
I20260321 02:57:20.625929 10202 raft_consensus.cc:3060] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.626128 10278 raft_consensus.cc:3055] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:57:20.626569 10278 raft_consensus.cc:740] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Running, Role: LEADER
I20260321 02:57:20.627628 10278 consensus_queue.cc:260] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.628989 10278 raft_consensus.cc:3060] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.631898 10202 raft_consensus.cc:2468] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 2.
I20260321 02:57:20.633344 10316 leader_election.cc:304] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 36fe4867413f4998b0aa2d13625f5edd; no voters:
I20260321 02:57:20.634136 10402 raft_consensus.cc:2804] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:57:20.634605 10402 raft_consensus.cc:697] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c [term 2 LEADER]: Becoming Leader. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Running, Role: LEADER
I20260321 02:57:20.635684 10402 consensus_queue.cc:237] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.637645 10278 raft_consensus.cc:2468] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 2.
I20260321 02:57:20.642452 10457 raft_consensus.cc:993] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c: : Instructing follower 44e26258105a4f16b7eb73d92056a7b5 to start an election
I20260321 02:57:20.642957 10402 raft_consensus.cc:1081] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Signalling peer 44e26258105a4f16b7eb73d92056a7b5 to start an election
I20260321 02:57:20.644794 10276 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1fc59958b093424e913602a709b24065"
dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
from {username='slave'} at 127.0.0.1:35800
I20260321 02:57:20.644188 10096 catalog_manager.cc:5671] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c reported cstate change: term changed from 1 to 2, leader changed from 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2) to 299a4a5db6eb447b86e68881e324911c (127.9.213.3). New cstate: current_term: 2 leader_uuid: "299a4a5db6eb447b86e68881e324911c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: HEALTHY } } }
I20260321 02:57:20.645314 10276 raft_consensus.cc:493] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:20.646154 10276 raft_consensus.cc:3060] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.652335 10276 raft_consensus.cc:515] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.653987 10276 leader_election.cc:290] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 2 election: Requested vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:20.655192 10202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fc59958b093424e913602a709b24065" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:20.655766 10202 raft_consensus.cc:3060] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.656208 10351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fc59958b093424e913602a709b24065" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "299a4a5db6eb447b86e68881e324911c"
I20260321 02:57:20.656847 10351 raft_consensus.cc:3055] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:57:20.657187 10351 raft_consensus.cc:740] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Running, Role: LEADER
I20260321 02:57:20.658064 10351 consensus_queue.cc:260] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.659585 10351 raft_consensus.cc:3060] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.661916 10202 raft_consensus.cc:2468] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 2.
I20260321 02:57:20.662981 10242 leader_election.cc:304] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36fe4867413f4998b0aa2d13625f5edd, 44e26258105a4f16b7eb73d92056a7b5; no voters:
I20260321 02:57:20.663869 10438 raft_consensus.cc:2804] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:57:20.664451 10438 raft_consensus.cc:697] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 LEADER]: Becoming Leader. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Running, Role: LEADER
I20260321 02:57:20.665278 10438 consensus_queue.cc:237] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:20.667030 10351 raft_consensus.cc:2468] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 2.
I20260321 02:57:20.677894 10096 catalog_manager.cc:5671] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 reported cstate change: term changed from 1 to 2, leader changed from 299a4a5db6eb447b86e68881e324911c (127.9.213.3) to 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2). New cstate: current_term: 2 leader_uuid: "44e26258105a4f16b7eb73d92056a7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:20.702471 10457 raft_consensus.cc:993] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c: : Instructing follower 36fe4867413f4998b0aa2d13625f5edd to start an election
I20260321 02:57:20.702908 10457 raft_consensus.cc:1081] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Signalling peer 36fe4867413f4998b0aa2d13625f5edd to start an election
I20260321 02:57:20.704828 10202 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b16dbc74e5dc4c4eb7e938db12adcfe5"
dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
from {username='slave'} at 127.0.0.1:45140
I20260321 02:57:20.705325 10202 raft_consensus.cc:493] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:20.705610 10202 raft_consensus.cc:3060] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.710563 10202 raft_consensus.cc:515] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } }
I20260321 02:57:20.712636 10202 leader_election.cc:290] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 2 election: Requested vote from peers 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003), 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025)
I20260321 02:57:20.713755 10351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b16dbc74e5dc4c4eb7e938db12adcfe5" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "299a4a5db6eb447b86e68881e324911c"
I20260321 02:57:20.713896 10276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b16dbc74e5dc4c4eb7e938db12adcfe5" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
I20260321 02:57:20.714480 10351 raft_consensus.cc:3055] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:57:20.714540 10276 raft_consensus.cc:3060] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.714934 10351 raft_consensus.cc:740] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Running, Role: LEADER
I20260321 02:57:20.715713 10351 consensus_queue.cc:260] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } }
I20260321 02:57:20.720240 10202 raft_consensus.cc:1275] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Refusing update from remote peer 44e26258105a4f16b7eb73d92056a7b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:20.720422 10351 raft_consensus.cc:3060] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.721764 10438 consensus_queue.cc:1048] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:20.725957 10276 raft_consensus.cc:2468] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fe4867413f4998b0aa2d13625f5edd in term 2.
I20260321 02:57:20.727154 10167 leader_election.cc:304] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36fe4867413f4998b0aa2d13625f5edd, 44e26258105a4f16b7eb73d92056a7b5; no voters:
I20260321 02:57:20.728127 10351 raft_consensus.cc:2468] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fe4867413f4998b0aa2d13625f5edd in term 2.
I20260321 02:57:20.728232 10405 raft_consensus.cc:2804] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:57:20.729152 10405 raft_consensus.cc:697] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 2 LEADER]: Becoming Leader. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Running, Role: LEADER
I20260321 02:57:20.730049 10405 consensus_queue.cc:237] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } }
I20260321 02:57:20.736573 10351 raft_consensus.cc:1275] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Refusing update from remote peer 44e26258105a4f16b7eb73d92056a7b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:20.741686 10461 consensus_queue.cc:1048] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:57:20.748346 10202 raft_consensus.cc:1275] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Refusing update from remote peer 44e26258105a4f16b7eb73d92056a7b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:20.749920 10480 consensus_queue.cc:1048] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:20.750180 10099 catalog_manager.cc:5671] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd reported cstate change: term changed from 1 to 2, leader changed from 299a4a5db6eb447b86e68881e324911c (127.9.213.3) to 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1). New cstate: current_term: 2 leader_uuid: "36fe4867413f4998b0aa2d13625f5edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: HEALTHY } } }
I20260321 02:57:20.759279 10352 raft_consensus.cc:1275] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Refusing update from remote peer 44e26258105a4f16b7eb73d92056a7b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:20.761126 10487 consensus_queue.cc:1048] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:20.943423 10276 raft_consensus.cc:1275] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Refusing update from remote peer 36fe4867413f4998b0aa2d13625f5edd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:20.945252 10405 consensus_queue.cc:1048] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:20.958631 10352 raft_consensus.cc:3060] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:20.969746 10351 raft_consensus.cc:1275] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Refusing update from remote peer 36fe4867413f4998b0aa2d13625f5edd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:20.972180 10405 consensus_queue.cc:1048] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:20.982573 10276 raft_consensus.cc:1275] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Refusing update from remote peer 36fe4867413f4998b0aa2d13625f5edd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:20.985090 10477 consensus_queue.cc:1048] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:57:20.996214 10351 raft_consensus.cc:1275] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Refusing update from remote peer 36fe4867413f4998b0aa2d13625f5edd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:20.997768 10489 consensus_queue.cc:1048] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:57:21.009590 10276 raft_consensus.cc:1275] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Refusing update from remote peer 36fe4867413f4998b0aa2d13625f5edd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:21.011723 10488 consensus_queue.cc:1048] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:57:21.017287 10352 raft_consensus.cc:1275] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Refusing update from remote peer 36fe4867413f4998b0aa2d13625f5edd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:21.019721 10488 consensus_queue.cc:1048] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:21.154768 10351 raft_consensus.cc:1275] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Refusing update from remote peer 44e26258105a4f16b7eb73d92056a7b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:21.156183 10438 consensus_queue.cc:1048] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:21.166769 10202 raft_consensus.cc:1275] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Refusing update from remote peer 44e26258105a4f16b7eb73d92056a7b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:21.168738 10438 consensus_queue.cc:1048] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:21.222332 10202 raft_consensus.cc:1275] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Refusing update from remote peer 299a4a5db6eb447b86e68881e324911c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:21.222438 10351 raft_consensus.cc:1275] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Refusing update from remote peer 36fe4867413f4998b0aa2d13625f5edd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:21.222946 10276 raft_consensus.cc:1275] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Refusing update from remote peer 36fe4867413f4998b0aa2d13625f5edd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:21.223930 10405 consensus_queue.cc:1048] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:57:21.224066 10457 consensus_queue.cc:1048] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:21.224725 10488 consensus_queue.cc:1048] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:21.255534 10276 raft_consensus.cc:1275] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Refusing update from remote peer 299a4a5db6eb447b86e68881e324911c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:21.257736 10500 consensus_queue.cc:1048] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:57:23.298352 10068 auto_leader_rebalancer-test.cc:272] The leader distribution is
I20260321 02:57:23.299299 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:23.324404 10203 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac"
dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
mode: GRACEFUL
new_leader_uuid: "299a4a5db6eb447b86e68881e324911c"
from {username='slave'} at 127.0.0.1:45170
I20260321 02:57:23.324975 10203 raft_consensus.cc:606] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 2 LEADER]: Received request to transfer leadership to TS 299a4a5db6eb447b86e68881e324911c
I20260321 02:57:23.336462 10276 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1fc59958b093424e913602a709b24065"
dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
mode: GRACEFUL
new_leader_uuid: "299a4a5db6eb447b86e68881e324911c"
from {username='slave'} at 127.0.0.1:35828
I20260321 02:57:23.336970 10276 raft_consensus.cc:606] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 LEADER]: Received request to transfer leadership to TS 299a4a5db6eb447b86e68881e324911c
I20260321 02:57:23.338056 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260321 02:57:23.338544 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:23.463194 10438 raft_consensus.cc:993] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5: : Instructing follower 299a4a5db6eb447b86e68881e324911c to start an election
I20260321 02:57:23.463711 10506 raft_consensus.cc:1081] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 LEADER]: Signalling peer 299a4a5db6eb447b86e68881e324911c to start an election
I20260321 02:57:23.465417 10351 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1fc59958b093424e913602a709b24065"
dest_uuid: "299a4a5db6eb447b86e68881e324911c"
from {username='slave'} at 127.0.0.1:34272
I20260321 02:57:23.465875 10351 raft_consensus.cc:493] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:23.466174 10351 raft_consensus.cc:3060] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:23.470043 10351 raft_consensus.cc:515] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:23.471261 10351 leader_election.cc:290] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 3 election: Requested vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025)
I20260321 02:57:23.472352 10203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fc59958b093424e913602a709b24065" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:23.472563 10276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fc59958b093424e913602a709b24065" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
I20260321 02:57:23.472849 10203 raft_consensus.cc:3060] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:23.473167 10276 raft_consensus.cc:3055] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:57:23.473471 10276 raft_consensus.cc:740] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Running, Role: LEADER
I20260321 02:57:23.474124 10276 consensus_queue.cc:260] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:23.475044 10276 raft_consensus.cc:3060] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:23.477602 10203 raft_consensus.cc:2468] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 3.
I20260321 02:57:23.478645 10316 leader_election.cc:304] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 36fe4867413f4998b0aa2d13625f5edd; no voters:
I20260321 02:57:23.479334 10504 raft_consensus.cc:2804] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:57:23.480300 10504 raft_consensus.cc:697] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 3 LEADER]: Becoming Leader. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Running, Role: LEADER
I20260321 02:57:23.481108 10504 consensus_queue.cc:237] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:23.481634 10276 raft_consensus.cc:2468] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 3.
I20260321 02:57:23.490262 10099 catalog_manager.cc:5671] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c reported cstate change: term changed from 2 to 3, leader changed from 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2) to 299a4a5db6eb447b86e68881e324911c (127.9.213.3). New cstate: current_term: 3 leader_uuid: "299a4a5db6eb447b86e68881e324911c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: HEALTHY } } }
I20260321 02:57:23.594333 10489 raft_consensus.cc:993] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd: : Instructing follower 299a4a5db6eb447b86e68881e324911c to start an election
I20260321 02:57:23.594710 10489 raft_consensus.cc:1081] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 2 LEADER]: Signalling peer 299a4a5db6eb447b86e68881e324911c to start an election
I20260321 02:57:23.597045 10351 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac"
dest_uuid: "299a4a5db6eb447b86e68881e324911c"
from {username='slave'} at 127.0.0.1:34280
I20260321 02:57:23.597595 10351 raft_consensus.cc:493] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:23.597906 10351 raft_consensus.cc:3060] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:23.601953 10351 raft_consensus.cc:515] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:23.603462 10351 leader_election.cc:290] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 3 election: Requested vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025)
I20260321 02:57:23.604666 10276 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
I20260321 02:57:23.604718 10203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:23.605479 10276 raft_consensus.cc:3060] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:23.605558 10203 raft_consensus.cc:3055] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:57:23.606022 10203 raft_consensus.cc:740] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Running, Role: LEADER
I20260321 02:57:23.606917 10203 consensus_queue.cc:260] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:23.608206 10203 raft_consensus.cc:3060] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:23.613016 10276 raft_consensus.cc:2468] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 3.
I20260321 02:57:23.614815 10316 leader_election.cc:304] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 44e26258105a4f16b7eb73d92056a7b5; no voters:
I20260321 02:57:23.615545 10504 raft_consensus.cc:2804] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:57:23.615808 10203 raft_consensus.cc:2468] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 3.
I20260321 02:57:23.616005 10504 raft_consensus.cc:697] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 3 LEADER]: Becoming Leader. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Running, Role: LEADER
I20260321 02:57:23.616931 10504 consensus_queue.cc:237] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:23.623662 10099 catalog_manager.cc:5671] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c reported cstate change: term changed from 2 to 3, leader changed from 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1) to 299a4a5db6eb447b86e68881e324911c (127.9.213.3). New cstate: current_term: 3 leader_uuid: "299a4a5db6eb447b86e68881e324911c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: HEALTHY } } }
I20260321 02:57:23.977731 10203 raft_consensus.cc:1275] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd [term 3 FOLLOWER]: Refusing update from remote peer 299a4a5db6eb447b86e68881e324911c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:57:23.979035 10504 consensus_queue.cc:1048] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:57:23.989908 10276 raft_consensus.cc:1275] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Refusing update from remote peer 299a4a5db6eb447b86e68881e324911c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:57:23.992463 10504 consensus_queue.cc:1048] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:57:24.029762 10276 raft_consensus.cc:1275] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Refusing update from remote peer 299a4a5db6eb447b86e68881e324911c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:57:24.031132 10504 consensus_queue.cc:1048] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:57:24.041664 10203 raft_consensus.cc:1275] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 3 FOLLOWER]: Refusing update from remote peer 299a4a5db6eb447b86e68881e324911c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:57:24.043282 10524 consensus_queue.cc:1048] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:57:24.339607 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:24.345232 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:24.345659 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:25.346554 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:25.351701 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:25.352133 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:26.353372 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:26.359500 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:26.360069 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:27.361166 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:27.365780 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:27.366209 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:28.367205 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:28.371805 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:28.372273 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:29.373323 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:29.377887 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:29.378321 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:30.379235 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:30.384009 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:30.384408 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:31.385315 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:31.390223 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:31.390671 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:32.391848 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:32.397032 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:32.397471 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:33.428877 10068 auto_leader_rebalancer-test.cc:291] The leader distribution is
I20260321 02:57:33.435274 10203 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b16dbc74e5dc4c4eb7e938db12adcfe5"
dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
mode: GRACEFUL
new_leader_uuid: "44e26258105a4f16b7eb73d92056a7b5"
from {username='slave'} at 127.0.0.1:45154
I20260321 02:57:33.436044 10203 raft_consensus.cc:606] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 2 LEADER]: Received request to transfer leadership to TS 44e26258105a4f16b7eb73d92056a7b5
I20260321 02:57:33.441141 10203 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5067bb4162bd4927839474abf3f445f3"
dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
mode: GRACEFUL
new_leader_uuid: "44e26258105a4f16b7eb73d92056a7b5"
from {username='slave'} at 127.0.0.1:45154
I20260321 02:57:33.441669 10203 raft_consensus.cc:606] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 2 LEADER]: Received request to transfer leadership to TS 44e26258105a4f16b7eb73d92056a7b5
I20260321 02:57:33.446980 10203 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7138ac120f3f4b0689010baea6f7cbc0"
dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
mode: GRACEFUL
new_leader_uuid: "44e26258105a4f16b7eb73d92056a7b5"
from {username='slave'} at 127.0.0.1:45154
I20260321 02:57:33.447475 10203 raft_consensus.cc:606] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 2 LEADER]: Received request to transfer leadership to TS 44e26258105a4f16b7eb73d92056a7b5
I20260321 02:57:33.454386 10351 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac"
dest_uuid: "299a4a5db6eb447b86e68881e324911c"
mode: GRACEFUL
new_leader_uuid: "44e26258105a4f16b7eb73d92056a7b5"
from {username='slave'} at 127.0.0.1:34282
I20260321 02:57:33.454885 10351 raft_consensus.cc:606] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 3 LEADER]: Received request to transfer leadership to TS 44e26258105a4f16b7eb73d92056a7b5
I20260321 02:57:33.460510 10351 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1fc59958b093424e913602a709b24065"
dest_uuid: "299a4a5db6eb447b86e68881e324911c"
mode: GRACEFUL
new_leader_uuid: "44e26258105a4f16b7eb73d92056a7b5"
from {username='slave'} at 127.0.0.1:34282
I20260321 02:57:33.461038 10351 raft_consensus.cc:606] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 3 LEADER]: Received request to transfer leadership to TS 44e26258105a4f16b7eb73d92056a7b5
I20260321 02:57:33.480382 10537 raft_consensus.cc:993] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c: : Instructing follower 44e26258105a4f16b7eb73d92056a7b5 to start an election
I20260321 02:57:33.480886 10537 raft_consensus.cc:1081] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 3 LEADER]: Signalling peer 44e26258105a4f16b7eb73d92056a7b5 to start an election
I20260321 02:57:33.482689 10278 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1fc59958b093424e913602a709b24065"
dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
from {username='slave'} at 127.0.0.1:35800
I20260321 02:57:33.483206 10278 raft_consensus.cc:493] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:33.483505 10278 raft_consensus.cc:3060] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:57:33.489631 10278 raft_consensus.cc:515] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:33.491401 10278 leader_election.cc:290] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 4 election: Requested vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:33.492558 10351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fc59958b093424e913602a709b24065" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "299a4a5db6eb447b86e68881e324911c"
I20260321 02:57:33.492329 10203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1fc59958b093424e913602a709b24065" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:33.493136 10351 raft_consensus.cc:3055] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:57:33.493305 10203 raft_consensus.cc:3060] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:57:33.493428 10351 raft_consensus.cc:740] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Running, Role: LEADER
I20260321 02:57:33.494349 10351 consensus_queue.cc:260] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:33.495543 10351 raft_consensus.cc:3060] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:57:33.498507 10203 raft_consensus.cc:2468] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 4.
I20260321 02:57:33.500115 10242 leader_election.cc:304] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36fe4867413f4998b0aa2d13625f5edd, 44e26258105a4f16b7eb73d92056a7b5; no voters:
I20260321 02:57:33.501103 10506 raft_consensus.cc:2804] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:57:33.502162 10506 raft_consensus.cc:697] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 4 LEADER]: Becoming Leader. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Running, Role: LEADER
I20260321 02:57:33.502004 10351 raft_consensus.cc:2468] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 4.
I20260321 02:57:33.503084 10506 consensus_queue.cc:237] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:33.510949 10099 catalog_manager.cc:5671] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 reported cstate change: term changed from 3 to 4, leader changed from 299a4a5db6eb447b86e68881e324911c (127.9.213.3) to 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2). New cstate: current_term: 4 leader_uuid: "44e26258105a4f16b7eb73d92056a7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:33.585685 10526 raft_consensus.cc:993] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd: : Instructing follower 44e26258105a4f16b7eb73d92056a7b5 to start an election
I20260321 02:57:33.586045 10526 raft_consensus.cc:1081] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 2 LEADER]: Signalling peer 44e26258105a4f16b7eb73d92056a7b5 to start an election
I20260321 02:57:33.588559 10278 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5067bb4162bd4927839474abf3f445f3"
dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
from {username='slave'} at 127.0.0.1:35802
I20260321 02:57:33.589058 10278 raft_consensus.cc:493] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:33.589356 10278 raft_consensus.cc:3060] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:33.596743 10278 raft_consensus.cc:515] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:33.598278 10278 leader_election.cc:290] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 3 election: Requested vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:33.599210 10203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5067bb4162bd4927839474abf3f445f3" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:33.599436 10351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5067bb4162bd4927839474abf3f445f3" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "299a4a5db6eb447b86e68881e324911c"
I20260321 02:57:33.599745 10203 raft_consensus.cc:3055] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:57:33.600189 10203 raft_consensus.cc:740] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Running, Role: LEADER
I20260321 02:57:33.600054 10351 raft_consensus.cc:3060] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:33.600919 10203 consensus_queue.cc:260] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:33.602037 10203 raft_consensus.cc:3060] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:33.605436 10351 raft_consensus.cc:2468] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 3.
I20260321 02:57:33.606750 10242 leader_election.cc:304] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 44e26258105a4f16b7eb73d92056a7b5; no voters:
I20260321 02:57:33.607877 10203 raft_consensus.cc:2468] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 3.
I20260321 02:57:33.607769 10506 raft_consensus.cc:2804] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:57:33.608788 10506 raft_consensus.cc:697] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 LEADER]: Becoming Leader. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Running, Role: LEADER
I20260321 02:57:33.609818 10506 consensus_queue.cc:237] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:33.618235 10099 catalog_manager.cc:5671] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 reported cstate change: term changed from 2 to 3, leader changed from 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1) to 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2). New cstate: current_term: 3 leader_uuid: "44e26258105a4f16b7eb73d92056a7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:33.804690 10532 raft_consensus.cc:993] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c: : Instructing follower 44e26258105a4f16b7eb73d92056a7b5 to start an election
I20260321 02:57:33.805193 10537 raft_consensus.cc:1081] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 3 LEADER]: Signalling peer 44e26258105a4f16b7eb73d92056a7b5 to start an election
I20260321 02:57:33.806667 10278 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac"
dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
from {username='slave'} at 127.0.0.1:35800
I20260321 02:57:33.807073 10278 raft_consensus.cc:493] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:33.807374 10278 raft_consensus.cc:3060] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:57:33.811331 10278 raft_consensus.cc:515] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:33.812605 10278 leader_election.cc:290] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 4 election: Requested vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:33.813782 10203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:33.813810 10351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "299a4a5db6eb447b86e68881e324911c"
I20260321 02:57:33.814471 10203 raft_consensus.cc:3060] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:57:33.814502 10351 raft_consensus.cc:3055] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:57:33.814899 10351 raft_consensus.cc:740] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Running, Role: LEADER
I20260321 02:57:33.815526 10351 consensus_queue.cc:260] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:33.816371 10351 raft_consensus.cc:3060] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:57:33.819180 10203 raft_consensus.cc:2468] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 4.
I20260321 02:57:33.820456 10242 leader_election.cc:304] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36fe4867413f4998b0aa2d13625f5edd, 44e26258105a4f16b7eb73d92056a7b5; no voters:
I20260321 02:57:33.821141 10351 raft_consensus.cc:2468] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 4.
I20260321 02:57:33.821288 10538 raft_consensus.cc:2804] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:57:33.821760 10538 raft_consensus.cc:697] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 4 LEADER]: Becoming Leader. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Running, Role: LEADER
I20260321 02:57:33.822533 10538 consensus_queue.cc:237] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:33.829422 10099 catalog_manager.cc:5671] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 reported cstate change: term changed from 3 to 4, leader changed from 299a4a5db6eb447b86e68881e324911c (127.9.213.3) to 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2). New cstate: current_term: 4 leader_uuid: "44e26258105a4f16b7eb73d92056a7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:33.840698 10544 raft_consensus.cc:993] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd: : Instructing follower 44e26258105a4f16b7eb73d92056a7b5 to start an election
I20260321 02:57:33.841156 10544 raft_consensus.cc:1081] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 2 LEADER]: Signalling peer 44e26258105a4f16b7eb73d92056a7b5 to start an election
I20260321 02:57:33.842757 10278 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b16dbc74e5dc4c4eb7e938db12adcfe5"
dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
from {username='slave'} at 127.0.0.1:35802
I20260321 02:57:33.843529 10278 raft_consensus.cc:493] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:33.843843 10278 raft_consensus.cc:3060] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:33.848706 10278 raft_consensus.cc:515] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } }
I20260321 02:57:33.850103 10278 leader_election.cc:290] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 3 election: Requested vote from peers 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003), 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771)
I20260321 02:57:33.851467 10203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b16dbc74e5dc4c4eb7e938db12adcfe5" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:33.851598 10351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b16dbc74e5dc4c4eb7e938db12adcfe5" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "299a4a5db6eb447b86e68881e324911c"
I20260321 02:57:33.852030 10203 raft_consensus.cc:3055] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:57:33.852373 10351 raft_consensus.cc:3060] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:33.852380 10203 raft_consensus.cc:740] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Running, Role: LEADER
I20260321 02:57:33.853557 10203 consensus_queue.cc:260] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } }
I20260321 02:57:33.854902 10203 raft_consensus.cc:3060] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:33.857123 10351 raft_consensus.cc:2468] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 3.
I20260321 02:57:33.858135 10242 leader_election.cc:304] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 44e26258105a4f16b7eb73d92056a7b5; no voters:
I20260321 02:57:33.859187 10538 raft_consensus.cc:2804] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:57:33.859602 10538 raft_consensus.cc:697] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 LEADER]: Becoming Leader. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Running, Role: LEADER
I20260321 02:57:33.860430 10203 raft_consensus.cc:2468] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 3.
I20260321 02:57:33.860433 10538 consensus_queue.cc:237] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } }
I20260321 02:57:33.868301 10099 catalog_manager.cc:5671] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 reported cstate change: term changed from 2 to 3, leader changed from 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1) to 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2). New cstate: current_term: 3 leader_uuid: "44e26258105a4f16b7eb73d92056a7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:33.938548 10526 raft_consensus.cc:993] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd: : Instructing follower 44e26258105a4f16b7eb73d92056a7b5 to start an election
I20260321 02:57:33.939018 10544 raft_consensus.cc:1081] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 2 LEADER]: Signalling peer 44e26258105a4f16b7eb73d92056a7b5 to start an election
I20260321 02:57:33.940672 10278 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7138ac120f3f4b0689010baea6f7cbc0"
dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
from {username='slave'} at 127.0.0.1:35802
I20260321 02:57:33.941372 10278 raft_consensus.cc:493] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:33.941658 10278 raft_consensus.cc:3060] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:33.946008 10278 raft_consensus.cc:515] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:33.947360 10278 leader_election.cc:290] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 3 election: Requested vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:33.948345 10203 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7138ac120f3f4b0689010baea6f7cbc0" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:33.948432 10351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7138ac120f3f4b0689010baea6f7cbc0" candidate_uuid: "44e26258105a4f16b7eb73d92056a7b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "299a4a5db6eb447b86e68881e324911c"
I20260321 02:57:33.948851 10203 raft_consensus.cc:3055] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:57:33.948997 10351 raft_consensus.cc:3060] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:33.949079 10203 raft_consensus.cc:740] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Running, Role: LEADER
I20260321 02:57:33.949846 10203 consensus_queue.cc:260] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:33.950675 10203 raft_consensus.cc:3060] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:33.955826 10351 raft_consensus.cc:2468] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 3.
I20260321 02:57:33.956815 10203 raft_consensus.cc:2468] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44e26258105a4f16b7eb73d92056a7b5 in term 3.
I20260321 02:57:33.957204 10242 leader_election.cc:304] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 44e26258105a4f16b7eb73d92056a7b5; no voters:
I20260321 02:57:33.957994 10506 raft_consensus.cc:2804] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:57:33.958439 10506 raft_consensus.cc:697] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 LEADER]: Becoming Leader. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Running, Role: LEADER
I20260321 02:57:33.959102 10506 consensus_queue.cc:237] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:33.965162 10099 catalog_manager.cc:5671] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 reported cstate change: term changed from 2 to 3, leader changed from 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1) to 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2). New cstate: current_term: 3 leader_uuid: "44e26258105a4f16b7eb73d92056a7b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:33.972898 10203 raft_consensus.cc:1275] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd [term 4 FOLLOWER]: Refusing update from remote peer 44e26258105a4f16b7eb73d92056a7b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 02:57:33.974114 10506 consensus_queue.cc:1048] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:57:33.985018 10351 raft_consensus.cc:1275] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 4 FOLLOWER]: Refusing update from remote peer 44e26258105a4f16b7eb73d92056a7b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 02:57:33.989351 10552 consensus_queue.cc:1048] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:57:34.063839 10351 raft_consensus.cc:1275] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Refusing update from remote peer 44e26258105a4f16b7eb73d92056a7b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:57:34.065160 10552 consensus_queue.cc:1048] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:57:34.074528 10203 raft_consensus.cc:1275] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 3 FOLLOWER]: Refusing update from remote peer 44e26258105a4f16b7eb73d92056a7b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:57:34.076207 10506 consensus_queue.cc:1048] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:57:34.252745 10203 raft_consensus.cc:1275] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 3 FOLLOWER]: Refusing update from remote peer 44e26258105a4f16b7eb73d92056a7b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:57:34.254460 10506 consensus_queue.cc:1048] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:57:34.259624 10202 raft_consensus.cc:1275] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 4 FOLLOWER]: Refusing update from remote peer 44e26258105a4f16b7eb73d92056a7b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 02:57:34.260967 10506 consensus_queue.cc:1048] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:57:34.268939 10351 raft_consensus.cc:1275] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Refusing update from remote peer 44e26258105a4f16b7eb73d92056a7b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:57:34.271174 10558 consensus_queue.cc:1048] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:57:34.276859 10352 raft_consensus.cc:1275] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 4 FOLLOWER]: Refusing update from remote peer 44e26258105a4f16b7eb73d92056a7b5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 02:57:34.280995 10506 consensus_queue.cc:1048] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:57:34.350567 10202 raft_consensus.cc:1275] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 3 FOLLOWER]: Refusing update from remote peer 44e26258105a4f16b7eb73d92056a7b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:57:34.351636 10506 consensus_queue.cc:1048] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:57:34.361914 10351 raft_consensus.cc:1275] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Refusing update from remote peer 44e26258105a4f16b7eb73d92056a7b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:57:34.365515 10506 consensus_queue.cc:1048] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:57:36.511338 10068 auto_leader_rebalancer-test.cc:306] The leader distribution is
I20260321 02:57:36.512235 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:36.520126 10278 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "02d438ffde12473b90a7549c34955b8e"
dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
mode: GRACEFUL
new_leader_uuid: "36fe4867413f4998b0aa2d13625f5edd"
from {username='slave'} at 127.0.0.1:35828
I20260321 02:57:36.520898 10278 raft_consensus.cc:606] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 2 LEADER]: Received request to transfer leadership to TS 36fe4867413f4998b0aa2d13625f5edd
I20260321 02:57:36.523681 10278 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac"
dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
mode: GRACEFUL
new_leader_uuid: "36fe4867413f4998b0aa2d13625f5edd"
from {username='slave'} at 127.0.0.1:35828
I20260321 02:57:36.524389 10278 raft_consensus.cc:606] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 4 LEADER]: Received request to transfer leadership to TS 36fe4867413f4998b0aa2d13625f5edd
I20260321 02:57:36.526928 10278 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5067bb4162bd4927839474abf3f445f3"
dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
mode: GRACEFUL
new_leader_uuid: "299a4a5db6eb447b86e68881e324911c"
from {username='slave'} at 127.0.0.1:35828
I20260321 02:57:36.527458 10278 raft_consensus.cc:606] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 LEADER]: Received request to transfer leadership to TS 299a4a5db6eb447b86e68881e324911c
I20260321 02:57:36.530352 10278 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7138ac120f3f4b0689010baea6f7cbc0"
dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
mode: GRACEFUL
new_leader_uuid: "299a4a5db6eb447b86e68881e324911c"
from {username='slave'} at 127.0.0.1:35828
I20260321 02:57:36.530884 10278 raft_consensus.cc:606] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 LEADER]: Received request to transfer leadership to TS 299a4a5db6eb447b86e68881e324911c
I20260321 02:57:36.533808 10278 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b16dbc74e5dc4c4eb7e938db12adcfe5"
dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
mode: GRACEFUL
new_leader_uuid: "36fe4867413f4998b0aa2d13625f5edd"
from {username='slave'} at 127.0.0.1:35828
I20260321 02:57:36.534364 10278 raft_consensus.cc:606] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 LEADER]: Received request to transfer leadership to TS 36fe4867413f4998b0aa2d13625f5edd
I20260321 02:57:36.535578 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260321 02:57:36.536252 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:36.657132 10538 raft_consensus.cc:993] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5: : Instructing follower 36fe4867413f4998b0aa2d13625f5edd to start an election
I20260321 02:57:36.657871 10552 raft_consensus.cc:1081] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 4 LEADER]: Signalling peer 36fe4867413f4998b0aa2d13625f5edd to start an election
I20260321 02:57:36.660634 10202 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac"
dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
from {username='slave'} at 127.0.0.1:45148
I20260321 02:57:36.661134 10202 raft_consensus.cc:493] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:36.661541 10202 raft_consensus.cc:3060] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:57:36.666785 10202 raft_consensus.cc:515] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:36.669898 10202 leader_election.cc:290] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 5 election: Requested vote from peers 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:36.671087 10278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
I20260321 02:57:36.671422 10351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c038d9ceb9f4044b53376479ddce5ac" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "299a4a5db6eb447b86e68881e324911c"
I20260321 02:57:36.672277 10278 raft_consensus.cc:3055] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:57:36.672551 10351 raft_consensus.cc:3060] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:57:36.672565 10278 raft_consensus.cc:740] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Running, Role: LEADER
I20260321 02:57:36.674306 10278 consensus_queue.cc:260] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:36.675956 10278 raft_consensus.cc:3060] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:57:36.679898 10351 raft_consensus.cc:2468] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fe4867413f4998b0aa2d13625f5edd in term 5.
I20260321 02:57:36.681500 10167 leader_election.cc:304] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 36fe4867413f4998b0aa2d13625f5edd; no voters:
I20260321 02:57:36.681998 10278 raft_consensus.cc:2468] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fe4867413f4998b0aa2d13625f5edd in term 5.
I20260321 02:57:36.683665 10567 raft_consensus.cc:2804] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:57:36.684841 10567 raft_consensus.cc:697] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 5 LEADER]: Becoming Leader. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Running, Role: LEADER
I20260321 02:57:36.685753 10567 consensus_queue.cc:237] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:36.692963 10096 catalog_manager.cc:5671] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd reported cstate change: term changed from 4 to 5, leader changed from 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2) to 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1). New cstate: current_term: 5 leader_uuid: "36fe4867413f4998b0aa2d13625f5edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:36.737164 10565 raft_consensus.cc:993] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5: : Instructing follower 299a4a5db6eb447b86e68881e324911c to start an election
I20260321 02:57:36.737650 10552 raft_consensus.cc:1081] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 LEADER]: Signalling peer 299a4a5db6eb447b86e68881e324911c to start an election
I20260321 02:57:36.739558 10351 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7138ac120f3f4b0689010baea6f7cbc0"
dest_uuid: "299a4a5db6eb447b86e68881e324911c"
from {username='slave'} at 127.0.0.1:34272
I20260321 02:57:36.740154 10351 raft_consensus.cc:493] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:36.740525 10351 raft_consensus.cc:3060] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:57:36.745851 10351 raft_consensus.cc:515] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:36.748612 10351 leader_election.cc:290] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 4 election: Requested vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025)
I20260321 02:57:36.749663 10202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7138ac120f3f4b0689010baea6f7cbc0" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:36.750233 10202 raft_consensus.cc:3060] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:57:36.750142 10278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7138ac120f3f4b0689010baea6f7cbc0" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
I20260321 02:57:36.750741 10278 raft_consensus.cc:3055] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:57:36.751053 10278 raft_consensus.cc:740] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Running, Role: LEADER
I20260321 02:57:36.752173 10278 consensus_queue.cc:260] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:36.753335 10278 raft_consensus.cc:3060] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:57:36.756843 10202 raft_consensus.cc:2468] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 4.
I20260321 02:57:36.758415 10316 leader_election.cc:304] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 36fe4867413f4998b0aa2d13625f5edd; no voters:
I20260321 02:57:36.759426 10566 raft_consensus.cc:2804] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:57:36.759805 10278 raft_consensus.cc:2468] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 4.
I20260321 02:57:36.760885 10566 raft_consensus.cc:697] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 4 LEADER]: Becoming Leader. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Running, Role: LEADER
I20260321 02:57:36.762061 10566 consensus_queue.cc:237] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:36.769450 10096 catalog_manager.cc:5671] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c reported cstate change: term changed from 3 to 4, leader changed from 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2) to 299a4a5db6eb447b86e68881e324911c (127.9.213.3). New cstate: current_term: 4 leader_uuid: "299a4a5db6eb447b86e68881e324911c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: HEALTHY } } }
I20260321 02:57:36.824278 10565 raft_consensus.cc:993] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5: : Instructing follower 299a4a5db6eb447b86e68881e324911c to start an election
I20260321 02:57:36.824712 10552 raft_consensus.cc:1081] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 LEADER]: Signalling peer 299a4a5db6eb447b86e68881e324911c to start an election
I20260321 02:57:36.826128 10351 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5067bb4162bd4927839474abf3f445f3"
dest_uuid: "299a4a5db6eb447b86e68881e324911c"
from {username='slave'} at 127.0.0.1:34272
I20260321 02:57:36.826558 10351 raft_consensus.cc:493] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:36.826809 10351 raft_consensus.cc:3060] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:57:36.831069 10351 raft_consensus.cc:515] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:36.832370 10351 leader_election.cc:290] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 4 election: Requested vote from peers 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771), 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025)
I20260321 02:57:36.833338 10202 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5067bb4162bd4927839474abf3f445f3" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
I20260321 02:57:36.833560 10278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5067bb4162bd4927839474abf3f445f3" candidate_uuid: "299a4a5db6eb447b86e68881e324911c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
I20260321 02:57:36.833905 10202 raft_consensus.cc:3060] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:57:36.834017 10278 raft_consensus.cc:3055] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:57:36.834327 10278 raft_consensus.cc:740] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Running, Role: LEADER
I20260321 02:57:36.835021 10278 consensus_queue.cc:260] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:36.836220 10278 raft_consensus.cc:3060] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:57:36.840222 10202 raft_consensus.cc:2468] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 4.
I20260321 02:57:36.841277 10316 leader_election.cc:304] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 36fe4867413f4998b0aa2d13625f5edd; no voters:
I20260321 02:57:36.842152 10566 raft_consensus.cc:2804] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:57:36.842437 10278 raft_consensus.cc:2468] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 299a4a5db6eb447b86e68881e324911c in term 4.
I20260321 02:57:36.842665 10566 raft_consensus.cc:697] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 4 LEADER]: Becoming Leader. State: Replica: 299a4a5db6eb447b86e68881e324911c, State: Running, Role: LEADER
I20260321 02:57:36.843605 10566 consensus_queue.cc:237] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:36.850311 10096 catalog_manager.cc:5671] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c reported cstate change: term changed from 3 to 4, leader changed from 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2) to 299a4a5db6eb447b86e68881e324911c (127.9.213.3). New cstate: current_term: 4 leader_uuid: "299a4a5db6eb447b86e68881e324911c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: HEALTHY } } }
I20260321 02:57:36.870090 10565 raft_consensus.cc:993] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5: : Instructing follower 36fe4867413f4998b0aa2d13625f5edd to start an election
I20260321 02:57:36.870761 10552 raft_consensus.cc:1081] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 LEADER]: Signalling peer 36fe4867413f4998b0aa2d13625f5edd to start an election
I20260321 02:57:36.872465 10202 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b16dbc74e5dc4c4eb7e938db12adcfe5"
dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
from {username='slave'} at 127.0.0.1:45148
I20260321 02:57:36.872934 10202 raft_consensus.cc:493] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:36.873277 10202 raft_consensus.cc:3060] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:57:36.877703 10202 raft_consensus.cc:515] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } }
I20260321 02:57:36.879132 10202 leader_election.cc:290] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 4 election: Requested vote from peers 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003), 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025)
I20260321 02:57:36.880229 10351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b16dbc74e5dc4c4eb7e938db12adcfe5" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "299a4a5db6eb447b86e68881e324911c"
I20260321 02:57:36.880501 10278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b16dbc74e5dc4c4eb7e938db12adcfe5" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
I20260321 02:57:36.880846 10351 raft_consensus.cc:3060] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:57:36.881044 10278 raft_consensus.cc:3055] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:57:36.881373 10278 raft_consensus.cc:740] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Running, Role: LEADER
I20260321 02:57:36.882045 10278 consensus_queue.cc:260] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } }
I20260321 02:57:36.882929 10278 raft_consensus.cc:3060] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:57:36.885885 10351 raft_consensus.cc:2468] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fe4867413f4998b0aa2d13625f5edd in term 4.
I20260321 02:57:36.887288 10167 leader_election.cc:304] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 36fe4867413f4998b0aa2d13625f5edd; no voters:
I20260321 02:57:36.887717 10278 raft_consensus.cc:2468] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fe4867413f4998b0aa2d13625f5edd in term 4.
I20260321 02:57:36.889814 10567 raft_consensus.cc:2804] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:57:36.890272 10567 raft_consensus.cc:697] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 4 LEADER]: Becoming Leader. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Running, Role: LEADER
I20260321 02:57:36.891404 10567 consensus_queue.cc:237] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } }
I20260321 02:57:36.899325 10096 catalog_manager.cc:5671] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd reported cstate change: term changed from 3 to 4, leader changed from 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2) to 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1). New cstate: current_term: 4 leader_uuid: "36fe4867413f4998b0aa2d13625f5edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: HEALTHY } } }
I20260321 02:57:37.002791 10565 raft_consensus.cc:993] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5: : Instructing follower 36fe4867413f4998b0aa2d13625f5edd to start an election
I20260321 02:57:37.003247 10552 raft_consensus.cc:1081] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 2 LEADER]: Signalling peer 36fe4867413f4998b0aa2d13625f5edd to start an election
I20260321 02:57:37.004669 10202 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "02d438ffde12473b90a7549c34955b8e"
dest_uuid: "36fe4867413f4998b0aa2d13625f5edd"
from {username='slave'} at 127.0.0.1:45148
I20260321 02:57:37.005136 10202 raft_consensus.cc:493] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:37.005425 10202 raft_consensus.cc:3060] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:37.010322 10202 raft_consensus.cc:515] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:37.012627 10202 leader_election.cc:290] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 3 election: Requested vote from peers 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2:42025), 299a4a5db6eb447b86e68881e324911c (127.9.213.3:39003)
I20260321 02:57:37.013563 10278 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02d438ffde12473b90a7549c34955b8e" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "44e26258105a4f16b7eb73d92056a7b5"
I20260321 02:57:37.014052 10351 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "02d438ffde12473b90a7549c34955b8e" candidate_uuid: "36fe4867413f4998b0aa2d13625f5edd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "299a4a5db6eb447b86e68881e324911c"
I20260321 02:57:37.014278 10278 raft_consensus.cc:3055] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:57:37.014585 10351 raft_consensus.cc:3060] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:37.014612 10278 raft_consensus.cc:740] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 44e26258105a4f16b7eb73d92056a7b5, State: Running, Role: LEADER
I20260321 02:57:37.015537 10278 consensus_queue.cc:260] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:37.016855 10278 raft_consensus.cc:3060] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:37.021729 10351 raft_consensus.cc:2468] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fe4867413f4998b0aa2d13625f5edd in term 3.
I20260321 02:57:37.022858 10167 leader_election.cc:304] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 299a4a5db6eb447b86e68881e324911c, 36fe4867413f4998b0aa2d13625f5edd; no voters:
I20260321 02:57:37.023301 10278 raft_consensus.cc:2468] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36fe4867413f4998b0aa2d13625f5edd in term 3.
I20260321 02:57:37.023869 10567 raft_consensus.cc:2804] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:57:37.024613 10567 raft_consensus.cc:697] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [term 3 LEADER]: Becoming Leader. State: Replica: 36fe4867413f4998b0aa2d13625f5edd, State: Running, Role: LEADER
I20260321 02:57:37.025405 10567 consensus_queue.cc:237] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } }
I20260321 02:57:37.033579 10096 catalog_manager.cc:5671] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd reported cstate change: term changed from 2 to 3, leader changed from 44e26258105a4f16b7eb73d92056a7b5 (127.9.213.2) to 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1). New cstate: current_term: 3 leader_uuid: "36fe4867413f4998b0aa2d13625f5edd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:37.210997 10351 raft_consensus.cc:1275] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 5 FOLLOWER]: Refusing update from remote peer 36fe4867413f4998b0aa2d13625f5edd: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260321 02:57:37.212469 10567 consensus_queue.cc:1048] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:57:37.223910 10278 raft_consensus.cc:1275] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 5 FOLLOWER]: Refusing update from remote peer 36fe4867413f4998b0aa2d13625f5edd: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260321 02:57:37.226006 10581 consensus_queue.cc:1048] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:57:37.272190 10278 raft_consensus.cc:1275] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 4 FOLLOWER]: Refusing update from remote peer 299a4a5db6eb447b86e68881e324911c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 02:57:37.273350 10566 consensus_queue.cc:1048] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:57:37.283319 10202 raft_consensus.cc:1275] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 4 FOLLOWER]: Refusing update from remote peer 299a4a5db6eb447b86e68881e324911c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 02:57:37.286931 10574 consensus_queue.cc:1048] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:57:37.335669 10351 raft_consensus.cc:1275] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 4 FOLLOWER]: Refusing update from remote peer 36fe4867413f4998b0aa2d13625f5edd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 02:57:37.336874 10581 consensus_queue.cc:1048] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:57:37.345927 10278 raft_consensus.cc:1275] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 4 FOLLOWER]: Refusing update from remote peer 36fe4867413f4998b0aa2d13625f5edd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 02:57:37.348266 10568 consensus_queue.cc:1048] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:57:37.374770 10202 raft_consensus.cc:1275] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 4 FOLLOWER]: Refusing update from remote peer 299a4a5db6eb447b86e68881e324911c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 02:57:37.374882 10278 raft_consensus.cc:1275] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 4 FOLLOWER]: Refusing update from remote peer 299a4a5db6eb447b86e68881e324911c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 02:57:37.376322 10574 consensus_queue.cc:1048] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Connected to new peer: Peer: permanent_uuid: "36fe4867413f4998b0aa2d13625f5edd" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:57:37.377381 10566 consensus_queue.cc:1048] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:57:37.537604 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:37.543426 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:37.543987 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:37.547040 10278 raft_consensus.cc:1275] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Refusing update from remote peer 36fe4867413f4998b0aa2d13625f5edd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:57:37.549057 10568 consensus_queue.cc:1048] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e26258105a4f16b7eb73d92056a7b5" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 42025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:57:37.560050 10351 raft_consensus.cc:1275] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Refusing update from remote peer 36fe4867413f4998b0aa2d13625f5edd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:57:37.561520 10568 consensus_queue.cc:1048] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [LEADER]: Connected to new peer: Peer: permanent_uuid: "299a4a5db6eb447b86e68881e324911c" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 39003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:57:38.545332 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:38.551033 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:38.551493 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:39.552644 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:39.558131 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:39.558593 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:40.559648 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:40.565868 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:40.566337 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:41.567795 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:41.572877 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:41.573355 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:42.574417 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:42.579219 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:42.579676 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:43.580771 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:43.585892 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:43.586370 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:44.587332 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:44.593827 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:44.594218 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:45.595197 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:45.600667 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:45.601019 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:46.632792 10068 auto_leader_rebalancer-test.cc:318] The leader distribution is
I20260321 02:57:46.633471 10068 tablet_server.cc:179] TabletServer@127.9.213.1:0 shutting down...
I20260321 02:57:46.670710 10068 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 02:57:46.671792 10068 tablet_replica.cc:333] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd: stopping tablet replica
I20260321 02:57:46.672820 10068 raft_consensus.cc:2243] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260321 02:57:46.673681 10068 raft_consensus.cc:2272] T f5b2778b93ef446c91564114305c7881 P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.677250 10068 tablet_replica.cc:333] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd: stopping tablet replica
I20260321 02:57:46.678036 10068 raft_consensus.cc:2243] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd [term 4 FOLLOWER]: Raft consensus shutting down.
I20260321 02:57:46.678694 10068 raft_consensus.cc:2272] T 1fc59958b093424e913602a709b24065 P 36fe4867413f4998b0aa2d13625f5edd [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.680909 10068 tablet_replica.cc:333] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd: stopping tablet replica
I20260321 02:57:46.681491 10068 raft_consensus.cc:2243] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [term 3 LEADER]: Raft consensus shutting down.
I20260321 02:57:46.682291 10068 raft_consensus.cc:2272] T 02d438ffde12473b90a7549c34955b8e P 36fe4867413f4998b0aa2d13625f5edd [term 3 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.684104 10068 tablet_replica.cc:333] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd: stopping tablet replica
I20260321 02:57:46.684584 10068 raft_consensus.cc:2243] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 5 LEADER]: Raft consensus shutting down.
I20260321 02:57:46.685343 10068 raft_consensus.cc:2272] T 0c038d9ceb9f4044b53376479ddce5ac P 36fe4867413f4998b0aa2d13625f5edd [term 5 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.686914 10068 tablet_replica.cc:333] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd: stopping tablet replica
I20260321 02:57:46.687399 10068 raft_consensus.cc:2243] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:57:46.687883 10068 raft_consensus.cc:2272] T 71c12424eea94c21a4aad2052dbac60e P 36fe4867413f4998b0aa2d13625f5edd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.689596 10068 tablet_replica.cc:333] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd: stopping tablet replica
I20260321 02:57:46.690119 10068 raft_consensus.cc:2243] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 4 LEADER]: Raft consensus shutting down.
I20260321 02:57:46.690877 10068 raft_consensus.cc:2272] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 36fe4867413f4998b0aa2d13625f5edd [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.692636 10068 tablet_replica.cc:333] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd: stopping tablet replica
I20260321 02:57:46.693102 10068 raft_consensus.cc:2243] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 4 FOLLOWER]: Raft consensus shutting down.
I20260321 02:57:46.693542 10068 raft_consensus.cc:2272] T 5067bb4162bd4927839474abf3f445f3 P 36fe4867413f4998b0aa2d13625f5edd [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.695842 10068 tablet_replica.cc:333] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd: stopping tablet replica
I20260321 02:57:46.696513 10068 raft_consensus.cc:2243] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 4 FOLLOWER]: Raft consensus shutting down.
I20260321 02:57:46.697052 10068 raft_consensus.cc:2272] T 7138ac120f3f4b0689010baea6f7cbc0 P 36fe4867413f4998b0aa2d13625f5edd [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.698802 10068 tablet_replica.cc:333] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd: stopping tablet replica
I20260321 02:57:46.699177 10068 raft_consensus.cc:2243] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260321 02:57:46.699520 10068 raft_consensus.cc:2272] T cb21c42d55ec496bb609b5ea5e6f844f P 36fe4867413f4998b0aa2d13625f5edd [term 2 FOLLOWER]: Raft consensus is shut down!
W20260321 02:57:46.806231 10242 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.9.213.1:46771: connect: Connection refused (error 111)
I20260321 02:57:46.812110 10068 tablet_server.cc:196] TabletServer@127.9.213.1:0 shutdown complete.
W20260321 02:57:46.817292 10316 consensus_peers.cc:597] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c -> Peer 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771): Couldn't send request to peer 36fe4867413f4998b0aa2d13625f5edd. Status: Network error: Client connection negotiation failed: client connection to 127.9.213.1:46771: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260321 02:57:46.820825 10316 consensus_peers.cc:597] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c -> Peer 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771): Couldn't send request to peer 36fe4867413f4998b0aa2d13625f5edd. Status: Network error: Client connection negotiation failed: client connection to 127.9.213.1:46771: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260321 02:57:46.821521 10242 consensus_peers.cc:597] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 -> Peer 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771): Couldn't send request to peer 36fe4867413f4998b0aa2d13625f5edd. Status: Network error: Client connection negotiation failed: client connection to 127.9.213.1:46771: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260321 02:57:46.829480 10316 consensus_peers.cc:597] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c -> Peer 36fe4867413f4998b0aa2d13625f5edd (127.9.213.1:46771): Couldn't send request to peer 36fe4867413f4998b0aa2d13625f5edd. Status: Network error: Client connection negotiation failed: client connection to 127.9.213.1:46771: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260321 02:57:46.838183 10068 tablet_server.cc:179] TabletServer@127.9.213.2:0 shutting down...
I20260321 02:57:46.870397 10068 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 02:57:46.871336 10068 tablet_replica.cc:333] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5: stopping tablet replica
I20260321 02:57:46.872298 10068 raft_consensus.cc:2243] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 LEADER]: Raft consensus shutting down.
I20260321 02:57:46.873448 10068 raft_consensus.cc:2272] T f5b2778b93ef446c91564114305c7881 P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.875352 10068 tablet_replica.cc:333] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5: stopping tablet replica
I20260321 02:57:46.875931 10068 raft_consensus.cc:2243] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260321 02:57:46.876497 10068 raft_consensus.cc:2272] T 0c038d9ceb9f4044b53376479ddce5ac P 44e26258105a4f16b7eb73d92056a7b5 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.878396 10068 tablet_replica.cc:333] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5: stopping tablet replica
I20260321 02:57:46.878948 10068 raft_consensus.cc:2243] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [term 1 LEADER]: Raft consensus shutting down.
I20260321 02:57:46.879752 10068 raft_consensus.cc:2272] T 71c12424eea94c21a4aad2052dbac60e P 44e26258105a4f16b7eb73d92056a7b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.881711 10068 tablet_replica.cc:333] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5: stopping tablet replica
I20260321 02:57:46.882300 10068 raft_consensus.cc:2243] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260321 02:57:46.882740 10068 raft_consensus.cc:2272] T 02d438ffde12473b90a7549c34955b8e P 44e26258105a4f16b7eb73d92056a7b5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.884542 10068 tablet_replica.cc:333] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5: stopping tablet replica
I20260321 02:57:46.885057 10068 raft_consensus.cc:2243] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260321 02:57:46.885524 10068 raft_consensus.cc:2272] T 7138ac120f3f4b0689010baea6f7cbc0 P 44e26258105a4f16b7eb73d92056a7b5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.887226 10068 tablet_replica.cc:333] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5: stopping tablet replica
I20260321 02:57:46.887715 10068 raft_consensus.cc:2243] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260321 02:57:46.888280 10068 raft_consensus.cc:2272] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 44e26258105a4f16b7eb73d92056a7b5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.890082 10068 tablet_replica.cc:333] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5: stopping tablet replica
I20260321 02:57:46.890579 10068 raft_consensus.cc:2243] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260321 02:57:46.890980 10068 raft_consensus.cc:2272] T 5067bb4162bd4927839474abf3f445f3 P 44e26258105a4f16b7eb73d92056a7b5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.893190 10068 tablet_replica.cc:333] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5: stopping tablet replica
I20260321 02:57:46.893728 10068 raft_consensus.cc:2243] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 4 LEADER]: Raft consensus shutting down.
I20260321 02:57:46.894977 10068 raft_consensus.cc:2272] T 1fc59958b093424e913602a709b24065 P 44e26258105a4f16b7eb73d92056a7b5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.897156 10068 tablet_replica.cc:333] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5: stopping tablet replica
I20260321 02:57:46.897686 10068 raft_consensus.cc:2243] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260321 02:57:46.898183 10068 raft_consensus.cc:2272] T cb21c42d55ec496bb609b5ea5e6f844f P 44e26258105a4f16b7eb73d92056a7b5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.927170 10068 tablet_server.cc:196] TabletServer@127.9.213.2:0 shutdown complete.
I20260321 02:57:46.946152 10068 tablet_server.cc:179] TabletServer@127.9.213.3:0 shutting down...
I20260321 02:57:46.981695 10068 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 02:57:46.982371 10068 tablet_replica.cc:333] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c: stopping tablet replica
I20260321 02:57:46.982993 10068 raft_consensus.cc:2243] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Raft consensus shutting down.
I20260321 02:57:46.983503 10068 raft_consensus.cc:2272] T 02d438ffde12473b90a7549c34955b8e P 299a4a5db6eb447b86e68881e324911c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.985630 10068 tablet_replica.cc:333] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c: stopping tablet replica
I20260321 02:57:46.986195 10068 raft_consensus.cc:2243] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c [term 2 LEADER]: Raft consensus shutting down.
I20260321 02:57:46.986994 10068 raft_consensus.cc:2272] T cb21c42d55ec496bb609b5ea5e6f844f P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.989593 10068 tablet_replica.cc:333] T 71c12424eea94c21a4aad2052dbac60e P 299a4a5db6eb447b86e68881e324911c: stopping tablet replica
I20260321 02:57:46.990358 10068 raft_consensus.cc:2243] T 71c12424eea94c21a4aad2052dbac60e P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:57:46.990824 10068 raft_consensus.cc:2272] T 71c12424eea94c21a4aad2052dbac60e P 299a4a5db6eb447b86e68881e324911c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.992519 10068 tablet_replica.cc:333] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c: stopping tablet replica
I20260321 02:57:46.993033 10068 raft_consensus.cc:2243] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 5 FOLLOWER]: Raft consensus shutting down.
I20260321 02:57:46.993491 10068 raft_consensus.cc:2272] T 0c038d9ceb9f4044b53376479ddce5ac P 299a4a5db6eb447b86e68881e324911c [term 5 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.995215 10068 tablet_replica.cc:333] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c: stopping tablet replica
I20260321 02:57:46.995693 10068 raft_consensus.cc:2243] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 4 FOLLOWER]: Raft consensus shutting down.
I20260321 02:57:46.996182 10068 raft_consensus.cc:2272] T 1fc59958b093424e913602a709b24065 P 299a4a5db6eb447b86e68881e324911c [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:46.997854 10068 tablet_replica.cc:333] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c: stopping tablet replica
I20260321 02:57:46.998346 10068 raft_consensus.cc:2243] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260321 02:57:46.998739 10068 raft_consensus.cc:2272] T f5b2778b93ef446c91564114305c7881 P 299a4a5db6eb447b86e68881e324911c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:47.000350 10068 tablet_replica.cc:333] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c: stopping tablet replica
I20260321 02:57:47.000818 10068 raft_consensus.cc:2243] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 4 LEADER]: Raft consensus shutting down.
I20260321 02:57:47.001696 10068 raft_consensus.cc:2272] T 5067bb4162bd4927839474abf3f445f3 P 299a4a5db6eb447b86e68881e324911c [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:47.003432 10068 tablet_replica.cc:333] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c: stopping tablet replica
I20260321 02:57:47.003903 10068 raft_consensus.cc:2243] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 4 FOLLOWER]: Raft consensus shutting down.
I20260321 02:57:47.004428 10068 raft_consensus.cc:2272] T b16dbc74e5dc4c4eb7e938db12adcfe5 P 299a4a5db6eb447b86e68881e324911c [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:47.006012 10068 tablet_replica.cc:333] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c: stopping tablet replica
I20260321 02:57:47.006471 10068 raft_consensus.cc:2243] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 4 LEADER]: Raft consensus shutting down.
I20260321 02:57:47.007236 10068 raft_consensus.cc:2272] T 7138ac120f3f4b0689010baea6f7cbc0 P 299a4a5db6eb447b86e68881e324911c [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:47.038405 10068 tablet_server.cc:196] TabletServer@127.9.213.3:0 shutdown complete.
I20260321 02:57:47.059091 10068 master.cc:562] Master@127.9.213.62:39551 shutting down...
I20260321 02:57:47.213716 10068 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5 [term 1 LEADER]: Raft consensus shutting down.
I20260321 02:57:47.214619 10068 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:57:47.214998 10068 tablet_replica.cc:333] T 00000000000000000000000000000000 P 7132a6305bb84eb3b9a755194521dfa5: stopping tablet replica
I20260321 02:57:47.234858 10068 master.cc:584] Master@127.9.213.62:39551 shutdown complete.
299a4a5db6eb447b86e68881e324911c 1
36fe4867413f4998b0aa2d13625f5edd 4
44e26258105a4f16b7eb73d92056a7b5 4
299a4a5db6eb447b86e68881e324911c 3
36fe4867413f4998b0aa2d13625f5edd 3
44e26258105a4f16b7eb73d92056a7b5 3
299a4a5db6eb447b86e68881e324911c 1
36fe4867413f4998b0aa2d13625f5edd 0
44e26258105a4f16b7eb73d92056a7b5 8
299a4a5db6eb447b86e68881e324911c 3
36fe4867413f4998b0aa2d13625f5edd 3
44e26258105a4f16b7eb73d92056a7b5 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31586 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260321 02:57:47.279726 10068 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.9.213.62:44005
I20260321 02:57:47.280926 10068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 02:57:47.286934 10606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 02:57:47.287673 10607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 02:57:47.288386 10609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 02:57:47.288667 10068 server_base.cc:1061] running on GCE node
I20260321 02:57:47.289841 10068 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 02:57:47.290102 10068 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 02:57:47.290220 10068 hybrid_clock.cc:648] HybridClock initialized: now 1774061867290209 us; error 0 us; skew 500 ppm
I20260321 02:57:47.290694 10068 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 02:57:47.294081 10068 webserver.cc:492] Webserver started at http://127.9.213.62:35639/ using document root <none> and password file <none>
I20260321 02:57:47.294598 10068 fs_manager.cc:362] Metadata directory not provided
I20260321 02:57:47.294777 10068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 02:57:47.295001 10068 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 02:57:47.296490 10068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1774061835490702-10068-0/minicluster-data/master-0-root/instance:
uuid: "c392fd13682945a19bbbf698726015f3"
format_stamp: "Formatted at 2026-03-21 02:57:47 on dist-test-slave-mz2z"
I20260321 02:57:47.301901 10068 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260321 02:57:47.306845 10614 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 02:57:47.307852 10068 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260321 02:57:47.308637 10068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1774061835490702-10068-0/minicluster-data/master-0-root
uuid: "c392fd13682945a19bbbf698726015f3"
format_stamp: "Formatted at 2026-03-21 02:57:47 on dist-test-slave-mz2z"
I20260321 02:57:47.309005 10068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1774061835490702-10068-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1774061835490702-10068-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1774061835490702-10068-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 02:57:47.321638 10068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 02:57:47.322911 10068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 02:57:47.363147 10068 rpc_server.cc:307] RPC server started. Bound to: 127.9.213.62:44005
I20260321 02:57:47.363277 10665 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.213.62:44005 every 8 connection(s)
I20260321 02:57:47.367069 10666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:47.378026 10666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3: Bootstrap starting.
I20260321 02:57:47.382833 10666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:47.387524 10666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3: No bootstrap required, opened a new log
I20260321 02:57:47.390534 10666 raft_consensus.cc:359] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c392fd13682945a19bbbf698726015f3" member_type: VOTER }
I20260321 02:57:47.391047 10666 raft_consensus.cc:385] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:47.391485 10666 raft_consensus.cc:740] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c392fd13682945a19bbbf698726015f3, State: Initialized, Role: FOLLOWER
I20260321 02:57:47.392145 10666 consensus_queue.cc:260] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c392fd13682945a19bbbf698726015f3" member_type: VOTER }
I20260321 02:57:47.392889 10666 raft_consensus.cc:399] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 02:57:47.393142 10666 raft_consensus.cc:493] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 02:57:47.393430 10666 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:47.397869 10666 raft_consensus.cc:515] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c392fd13682945a19bbbf698726015f3" member_type: VOTER }
I20260321 02:57:47.398478 10666 leader_election.cc:304] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c392fd13682945a19bbbf698726015f3; no voters:
I20260321 02:57:47.400009 10666 leader_election.cc:290] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260321 02:57:47.400419 10669 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:47.402007 10669 raft_consensus.cc:697] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3 [term 1 LEADER]: Becoming Leader. State: Replica: c392fd13682945a19bbbf698726015f3, State: Running, Role: LEADER
I20260321 02:57:47.402802 10669 consensus_queue.cc:237] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c392fd13682945a19bbbf698726015f3" member_type: VOTER }
I20260321 02:57:47.403820 10666 sys_catalog.cc:565] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260321 02:57:47.409427 10670 sys_catalog.cc:455] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c392fd13682945a19bbbf698726015f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c392fd13682945a19bbbf698726015f3" member_type: VOTER } }
I20260321 02:57:47.409556 10671 sys_catalog.cc:455] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c392fd13682945a19bbbf698726015f3. Latest consensus state: current_term: 1 leader_uuid: "c392fd13682945a19bbbf698726015f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c392fd13682945a19bbbf698726015f3" member_type: VOTER } }
I20260321 02:57:47.410238 10671 sys_catalog.cc:458] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3 [sys.catalog]: This master's current role is: LEADER
I20260321 02:57:47.410156 10670 sys_catalog.cc:458] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3 [sys.catalog]: This master's current role is: LEADER
I20260321 02:57:47.418612 10676 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260321 02:57:47.423377 10676 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260321 02:57:47.425889 10068 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260321 02:57:47.433152 10676 catalog_manager.cc:1357] Generated new cluster ID: 93369f573b0d4fdf9f1b125b8de4495b
I20260321 02:57:47.433524 10676 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260321 02:57:47.447336 10676 catalog_manager.cc:1380] Generated new certificate authority record
I20260321 02:57:47.448691 10676 catalog_manager.cc:1514] Loading token signing keys...
I20260321 02:57:47.461654 10676 catalog_manager.cc:6044] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3: Generated new TSK 0
I20260321 02:57:47.462260 10676 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260321 02:57:47.493070 10068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 02:57:47.500748 10687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 02:57:47.501585 10688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 02:57:47.503826 10690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 02:57:47.503865 10068 server_base.cc:1061] running on GCE node
I20260321 02:57:47.505295 10068 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 02:57:47.505528 10068 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 02:57:47.505700 10068 hybrid_clock.cc:648] HybridClock initialized: now 1774061867505688 us; error 0 us; skew 500 ppm
I20260321 02:57:47.506430 10068 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 02:57:47.509109 10068 webserver.cc:492] Webserver started at http://127.9.213.1:33247/ using document root <none> and password file <none>
I20260321 02:57:47.509665 10068 fs_manager.cc:362] Metadata directory not provided
I20260321 02:57:47.509856 10068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 02:57:47.510129 10068 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 02:57:47.511644 10068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1774061835490702-10068-0/minicluster-data/ts-0-root/instance:
uuid: "153c2de7193c4844b765382a198e9db2"
format_stamp: "Formatted at 2026-03-21 02:57:47 on dist-test-slave-mz2z"
I20260321 02:57:47.516925 10068 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260321 02:57:47.520792 10695 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 02:57:47.521739 10068 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260321 02:57:47.522083 10068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1774061835490702-10068-0/minicluster-data/ts-0-root
uuid: "153c2de7193c4844b765382a198e9db2"
format_stamp: "Formatted at 2026-03-21 02:57:47 on dist-test-slave-mz2z"
I20260321 02:57:47.522408 10068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1774061835490702-10068-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1774061835490702-10068-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1774061835490702-10068-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 02:57:47.579545 10068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 02:57:47.580978 10068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 02:57:47.582581 10068 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 02:57:47.584977 10068 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 02:57:47.585242 10068 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 02:57:47.585536 10068 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 02:57:47.585740 10068 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 02:57:47.630631 10068 rpc_server.cc:307] RPC server started. Bound to: 127.9.213.1:42123
I20260321 02:57:47.630750 10757 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.213.1:42123 every 8 connection(s)
I20260321 02:57:47.636106 10068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 02:57:47.642961 10762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 02:57:47.644049 10763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 02:57:47.646466 10068 server_base.cc:1061] running on GCE node
W20260321 02:57:47.648387 10765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 02:57:47.649410 10068 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 02:57:47.649659 10068 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 02:57:47.649833 10068 hybrid_clock.cc:648] HybridClock initialized: now 1774061867649815 us; error 0 us; skew 500 ppm
I20260321 02:57:47.650382 10068 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 02:57:47.653523 10068 webserver.cc:492] Webserver started at http://127.9.213.2:34095/ using document root <none> and password file <none>
I20260321 02:57:47.654105 10068 fs_manager.cc:362] Metadata directory not provided
I20260321 02:57:47.654320 10068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 02:57:47.654755 10068 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 02:57:47.654742 10758 heartbeater.cc:344] Connected to a master server at 127.9.213.62:44005
I20260321 02:57:47.655390 10758 heartbeater.cc:461] Registering TS with master...
I20260321 02:57:47.656133 10758 heartbeater.cc:507] Master 127.9.213.62:44005 requested a full tablet report, sending...
I20260321 02:57:47.656262 10068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1774061835490702-10068-0/minicluster-data/ts-1-root/instance:
uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
format_stamp: "Formatted at 2026-03-21 02:57:47 on dist-test-slave-mz2z"
I20260321 02:57:47.658165 10631 ts_manager.cc:194] Registered new tserver with Master: 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123)
I20260321 02:57:47.659878 10631 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59284
I20260321 02:57:47.661171 10068 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260321 02:57:47.664848 10770 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 02:57:47.665764 10068 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260321 02:57:47.666008 10068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1774061835490702-10068-0/minicluster-data/ts-1-root
uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
format_stamp: "Formatted at 2026-03-21 02:57:47 on dist-test-slave-mz2z"
I20260321 02:57:47.666246 10068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1774061835490702-10068-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1774061835490702-10068-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1774061835490702-10068-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 02:57:47.690935 10068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 02:57:47.692283 10068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 02:57:47.693806 10068 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 02:57:47.696353 10068 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 02:57:47.696548 10068 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 02:57:47.696801 10068 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 02:57:47.697103 10068 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 02:57:47.742273 10068 rpc_server.cc:307] RPC server started. Bound to: 127.9.213.2:44303
I20260321 02:57:47.742416 10832 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.213.2:44303 every 8 connection(s)
I20260321 02:57:47.747149 10068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 02:57:47.755080 10836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 02:57:47.755775 10837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 02:57:47.759802 10068 server_base.cc:1061] running on GCE node
W20260321 02:57:47.760553 10839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 02:57:47.761026 10833 heartbeater.cc:344] Connected to a master server at 127.9.213.62:44005
I20260321 02:57:47.761648 10833 heartbeater.cc:461] Registering TS with master...
I20260321 02:57:47.761834 10068 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 02:57:47.762089 10068 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 02:57:47.762449 10068 hybrid_clock.cc:648] HybridClock initialized: now 1774061867762280 us; error 0 us; skew 500 ppm
I20260321 02:57:47.762495 10833 heartbeater.cc:507] Master 127.9.213.62:44005 requested a full tablet report, sending...
I20260321 02:57:47.763128 10068 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 02:57:47.764906 10631 ts_manager.cc:194] Registered new tserver with Master: 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303)
I20260321 02:57:47.766789 10068 webserver.cc:492] Webserver started at http://127.9.213.3:42081/ using document root <none> and password file <none>
I20260321 02:57:47.767704 10068 fs_manager.cc:362] Metadata directory not provided
I20260321 02:57:47.767683 10631 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59300
I20260321 02:57:47.768219 10068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 02:57:47.768661 10068 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 02:57:47.769956 10068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1774061835490702-10068-0/minicluster-data/ts-2-root/instance:
uuid: "5ed785fa1622433e8993e90b8c6c984e"
format_stamp: "Formatted at 2026-03-21 02:57:47 on dist-test-slave-mz2z"
I20260321 02:57:47.774842 10068 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260321 02:57:47.779122 10844 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 02:57:47.780263 10068 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260321 02:57:47.780639 10068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1774061835490702-10068-0/minicluster-data/ts-2-root
uuid: "5ed785fa1622433e8993e90b8c6c984e"
format_stamp: "Formatted at 2026-03-21 02:57:47 on dist-test-slave-mz2z"
I20260321 02:57:47.780947 10068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1774061835490702-10068-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1774061835490702-10068-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1774061835490702-10068-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 02:57:47.812075 10068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 02:57:47.813490 10068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 02:57:47.814888 10068 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 02:57:47.817173 10068 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 02:57:47.817406 10068 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 02:57:47.817646 10068 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 02:57:47.817821 10068 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 02:57:47.862959 10068 rpc_server.cc:307] RPC server started. Bound to: 127.9.213.3:34785
I20260321 02:57:47.863126 10906 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.213.3:34785 every 8 connection(s)
I20260321 02:57:47.877882 10907 heartbeater.cc:344] Connected to a master server at 127.9.213.62:44005
I20260321 02:57:47.878252 10907 heartbeater.cc:461] Registering TS with master...
I20260321 02:57:47.879037 10907 heartbeater.cc:507] Master 127.9.213.62:44005 requested a full tablet report, sending...
I20260321 02:57:47.881243 10631 ts_manager.cc:194] Registered new tserver with Master: 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:47.882531 10068 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016088146s
I20260321 02:57:47.882650 10631 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59314
I20260321 02:57:47.911875 10631 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59330:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260321 02:57:47.916527 10631 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260321 02:57:47.995621 10723 tablet_service.cc:1511] Processing CreateTablet for tablet 6e161a85964742b09837e0ae3dda87ca (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260321 02:57:47.997309 10723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e161a85964742b09837e0ae3dda87ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.005251 10798 tablet_service.cc:1511] Processing CreateTablet for tablet 6e161a85964742b09837e0ae3dda87ca (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260321 02:57:48.005339 10721 tablet_service.cc:1511] Processing CreateTablet for tablet 08f9bfdb4a464783b9b5b88025e96c4b (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260321 02:57:48.005642 10720 tablet_service.cc:1511] Processing CreateTablet for tablet 3a1c86bd318d452e9446f7ab8783c20f (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260321 02:57:48.006918 10798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e161a85964742b09837e0ae3dda87ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.007055 10721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08f9bfdb4a464783b9b5b88025e96c4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.007368 10795 tablet_service.cc:1511] Processing CreateTablet for tablet 3a1c86bd318d452e9446f7ab8783c20f (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260321 02:57:48.008292 10722 tablet_service.cc:1511] Processing CreateTablet for tablet 082582ef8a7a483eaf98b4ced6214a57 (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260321 02:57:48.008802 10795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a1c86bd318d452e9446f7ab8783c20f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.008824 10719 tablet_service.cc:1511] Processing CreateTablet for tablet f35df046e02643dab408161cee9f00a4 (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260321 02:57:48.009702 10722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 082582ef8a7a483eaf98b4ced6214a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.010347 10719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f35df046e02643dab408161cee9f00a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.005466 10794 tablet_service.cc:1511] Processing CreateTablet for tablet f35df046e02643dab408161cee9f00a4 (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260321 02:57:48.011601 10794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f35df046e02643dab408161cee9f00a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.012523 10720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a1c86bd318d452e9446f7ab8783c20f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.012658 10796 tablet_service.cc:1511] Processing CreateTablet for tablet 08f9bfdb4a464783b9b5b88025e96c4b (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260321 02:57:48.014058 10796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08f9bfdb4a464783b9b5b88025e96c4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.016868 10797 tablet_service.cc:1511] Processing CreateTablet for tablet 082582ef8a7a483eaf98b4ced6214a57 (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260321 02:57:48.018266 10797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 082582ef8a7a483eaf98b4ced6214a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.022339 10718 tablet_service.cc:1511] Processing CreateTablet for tablet a56d5d5429b743bc8d2cb6fcb999f86e (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260321 02:57:48.023991 10718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a56d5d5429b743bc8d2cb6fcb999f86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.038834 10922 tablet_bootstrap.cc:492] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2: Bootstrap starting.
I20260321 02:57:48.044551 10793 tablet_service.cc:1511] Processing CreateTablet for tablet a56d5d5429b743bc8d2cb6fcb999f86e (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260321 02:57:48.047811 10922 tablet_bootstrap.cc:654] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.048089 10792 tablet_service.cc:1511] Processing CreateTablet for tablet e1d08251e74b419599f24a910bc9e538 (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260321 02:57:48.050604 10872 tablet_service.cc:1511] Processing CreateTablet for tablet 6e161a85964742b09837e0ae3dda87ca (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260321 02:57:48.052475 10871 tablet_service.cc:1511] Processing CreateTablet for tablet 082582ef8a7a483eaf98b4ced6214a57 (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260321 02:57:48.053207 10870 tablet_service.cc:1511] Processing CreateTablet for tablet 08f9bfdb4a464783b9b5b88025e96c4b (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260321 02:57:48.054840 10723 tablet_service.cc:1511] Processing CreateTablet for tablet e1d08251e74b419599f24a910bc9e538 (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260321 02:57:48.055567 10869 tablet_service.cc:1511] Processing CreateTablet for tablet 3a1c86bd318d452e9446f7ab8783c20f (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260321 02:57:48.056229 10723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1d08251e74b419599f24a910bc9e538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.057940 10868 tablet_service.cc:1511] Processing CreateTablet for tablet f35df046e02643dab408161cee9f00a4 (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260321 02:57:48.061256 10867 tablet_service.cc:1511] Processing CreateTablet for tablet a56d5d5429b743bc8d2cb6fcb999f86e (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260321 02:57:48.046286 10793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a56d5d5429b743bc8d2cb6fcb999f86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.066980 10866 tablet_service.cc:1511] Processing CreateTablet for tablet e1d08251e74b419599f24a910bc9e538 (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260321 02:57:48.069123 10872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e161a85964742b09837e0ae3dda87ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.075775 10791 tablet_service.cc:1511] Processing CreateTablet for tablet b2633115df76454eb4c74d33d2b0b8fe (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260321 02:57:48.076366 10869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a1c86bd318d452e9446f7ab8783c20f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.069743 10792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1d08251e74b419599f24a910bc9e538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.094089 10865 tablet_service.cc:1511] Processing CreateTablet for tablet b2633115df76454eb4c74d33d2b0b8fe (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260321 02:57:48.100919 10719 tablet_service.cc:1511] Processing CreateTablet for tablet b2633115df76454eb4c74d33d2b0b8fe (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260321 02:57:48.101388 10718 tablet_service.cc:1511] Processing CreateTablet for tablet 8619a3b4cc1b4d59b5a312ac0bc01017 (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260321 02:57:48.102885 10719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2633115df76454eb4c74d33d2b0b8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.103799 10718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8619a3b4cc1b4d59b5a312ac0bc01017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.104542 10870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08f9bfdb4a464783b9b5b88025e96c4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.105327 10868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f35df046e02643dab408161cee9f00a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.106905 10867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a56d5d5429b743bc8d2cb6fcb999f86e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.116357 10791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2633115df76454eb4c74d33d2b0b8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.121241 10798 tablet_service.cc:1511] Processing CreateTablet for tablet 8619a3b4cc1b4d59b5a312ac0bc01017 (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260321 02:57:48.122789 10798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8619a3b4cc1b4d59b5a312ac0bc01017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.128432 10865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2633115df76454eb4c74d33d2b0b8fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.128979 10864 tablet_service.cc:1511] Processing CreateTablet for tablet 8619a3b4cc1b4d59b5a312ac0bc01017 (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260321 02:57:48.130844 10864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8619a3b4cc1b4d59b5a312ac0bc01017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.132367 10866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1d08251e74b419599f24a910bc9e538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.136371 10871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 082582ef8a7a483eaf98b4ced6214a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.142149 10923 tablet_bootstrap.cc:492] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8: Bootstrap starting.
I20260321 02:57:48.151422 10923 tablet_bootstrap.cc:654] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.190248 10863 tablet_service.cc:1511] Processing CreateTablet for tablet 60d9453110c241f4aaa3740c3a43e6b6 (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260321 02:57:48.191532 10922 tablet_bootstrap.cc:492] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2: No bootstrap required, opened a new log
I20260321 02:57:48.191732 10863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60d9453110c241f4aaa3740c3a43e6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.200963 10797 tablet_service.cc:1511] Processing CreateTablet for tablet 60d9453110c241f4aaa3740c3a43e6b6 (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260321 02:57:48.202622 10797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60d9453110c241f4aaa3740c3a43e6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.200763 10722 tablet_service.cc:1511] Processing CreateTablet for tablet 60d9453110c241f4aaa3740c3a43e6b6 (DEFAULT_TABLE table=test-workload [id=cbef4a1da0ca4e379513c5abc62378c6]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260321 02:57:48.205628 10722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60d9453110c241f4aaa3740c3a43e6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:57:48.212719 10925 tablet_bootstrap.cc:492] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e: Bootstrap starting.
I20260321 02:57:48.192032 10922 ts_tablet_manager.cc:1403] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2: Time spent bootstrapping tablet: real 0.154s user 0.062s sys 0.052s
I20260321 02:57:48.224896 10925 tablet_bootstrap.cc:654] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.234970 10922 raft_consensus.cc:359] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.235819 10922 raft_consensus.cc:385] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.236172 10922 raft_consensus.cc:740] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.236896 10922 consensus_queue.cc:260] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.248325 10758 heartbeater.cc:499] Master 127.9.213.62:44005 was elected leader, sending a full tablet report...
I20260321 02:57:48.248916 10922 ts_tablet_manager.cc:1434] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20260321 02:57:48.249891 10922 tablet_bootstrap.cc:492] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2: Bootstrap starting.
I20260321 02:57:48.253651 10923 tablet_bootstrap.cc:492] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8: No bootstrap required, opened a new log
I20260321 02:57:48.254140 10923 ts_tablet_manager.cc:1403] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8: Time spent bootstrapping tablet: real 0.112s user 0.018s sys 0.013s
I20260321 02:57:48.259670 10922 tablet_bootstrap.cc:654] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.259661 10923 raft_consensus.cc:359] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.264811 10923 raft_consensus.cc:385] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.265172 10923 raft_consensus.cc:740] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.265898 10923 consensus_queue.cc:260] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.267237 10925 tablet_bootstrap.cc:492] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e: No bootstrap required, opened a new log
I20260321 02:57:48.267787 10925 ts_tablet_manager.cc:1403] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e: Time spent bootstrapping tablet: real 0.055s user 0.010s sys 0.005s
I20260321 02:57:48.269049 10833 heartbeater.cc:499] Master 127.9.213.62:44005 was elected leader, sending a full tablet report...
I20260321 02:57:48.269706 10923 ts_tablet_manager.cc:1434] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8: Time spent starting tablet: real 0.015s user 0.005s sys 0.005s
I20260321 02:57:48.270637 10923 tablet_bootstrap.cc:492] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8: Bootstrap starting.
I20260321 02:57:48.272786 10925 raft_consensus.cc:359] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.273567 10925 raft_consensus.cc:385] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.273860 10925 raft_consensus.cc:740] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.274654 10925 consensus_queue.cc:260] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.277387 10923 tablet_bootstrap.cc:654] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.281528 10922 tablet_bootstrap.cc:492] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2: No bootstrap required, opened a new log
I20260321 02:57:48.282079 10922 ts_tablet_manager.cc:1403] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.003s
I20260321 02:57:48.285254 10922 raft_consensus.cc:359] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.286249 10922 raft_consensus.cc:385] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.286617 10922 raft_consensus.cc:740] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.287587 10907 heartbeater.cc:499] Master 127.9.213.62:44005 was elected leader, sending a full tablet report...
I20260321 02:57:48.287601 10922 consensus_queue.cc:260] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.290640 10922 ts_tablet_manager.cc:1434] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260321 02:57:48.294179 10925 ts_tablet_manager.cc:1434] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e: Time spent starting tablet: real 0.026s user 0.014s sys 0.003s
I20260321 02:57:48.295326 10925 tablet_bootstrap.cc:492] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e: Bootstrap starting.
I20260321 02:57:48.295432 10922 tablet_bootstrap.cc:492] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2: Bootstrap starting.
I20260321 02:57:48.298197 10923 tablet_bootstrap.cc:492] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8: No bootstrap required, opened a new log
I20260321 02:57:48.298828 10923 ts_tablet_manager.cc:1403] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.007s
I20260321 02:57:48.301785 10923 raft_consensus.cc:359] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.302505 10925 tablet_bootstrap.cc:654] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.302837 10923 raft_consensus.cc:385] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.303236 10923 raft_consensus.cc:740] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.302285 10922 tablet_bootstrap.cc:654] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.304246 10923 consensus_queue.cc:260] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.312053 10923 ts_tablet_manager.cc:1434] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8: Time spent starting tablet: real 0.013s user 0.007s sys 0.001s
I20260321 02:57:48.326604 10923 tablet_bootstrap.cc:492] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8: Bootstrap starting.
I20260321 02:57:48.345430 10923 tablet_bootstrap.cc:654] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.349376 10922 tablet_bootstrap.cc:492] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2: No bootstrap required, opened a new log
I20260321 02:57:48.349738 10925 tablet_bootstrap.cc:492] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e: No bootstrap required, opened a new log
I20260321 02:57:48.350019 10922 ts_tablet_manager.cc:1403] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.035s
I20260321 02:57:48.350262 10925 ts_tablet_manager.cc:1403] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e: Time spent bootstrapping tablet: real 0.055s user 0.012s sys 0.042s
I20260321 02:57:48.352591 10923 tablet_bootstrap.cc:492] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8: No bootstrap required, opened a new log
I20260321 02:57:48.352586 10925 raft_consensus.cc:359] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.353199 10923 ts_tablet_manager.cc:1403] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.004s
I20260321 02:57:48.352967 10922 raft_consensus.cc:359] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.353441 10925 raft_consensus.cc:385] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.353670 10922 raft_consensus.cc:385] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.353765 10925 raft_consensus.cc:740] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.354027 10922 raft_consensus.cc:740] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.354664 10925 consensus_queue.cc:260] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.354756 10922 consensus_queue.cc:260] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.355815 10923 raft_consensus.cc:359] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.356535 10923 raft_consensus.cc:385] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.356833 10925 ts_tablet_manager.cc:1434] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260321 02:57:48.356810 10923 raft_consensus.cc:740] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.357204 10922 ts_tablet_manager.cc:1434] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2: Time spent starting tablet: real 0.007s user 0.009s sys 0.000s
I20260321 02:57:48.357857 10925 tablet_bootstrap.cc:492] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e: Bootstrap starting.
I20260321 02:57:48.358175 10922 tablet_bootstrap.cc:492] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2: Bootstrap starting.
I20260321 02:57:48.357815 10923 consensus_queue.cc:260] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.364310 10923 ts_tablet_manager.cc:1434] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260321 02:57:48.364858 10922 tablet_bootstrap.cc:654] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.365123 10925 tablet_bootstrap.cc:654] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.365077 10930 raft_consensus.cc:493] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:48.365655 10930 raft_consensus.cc:515] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.371588 10923 tablet_bootstrap.cc:492] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8: Bootstrap starting.
I20260321 02:57:48.373023 10930 leader_election.cc:290] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:48.378556 10922 tablet_bootstrap.cc:492] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2: No bootstrap required, opened a new log
I20260321 02:57:48.379930 10922 ts_tablet_manager.cc:1403] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2: Time spent bootstrapping tablet: real 0.022s user 0.018s sys 0.000s
I20260321 02:57:48.383759 10922 raft_consensus.cc:359] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.384552 10922 raft_consensus.cc:385] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.384862 10922 raft_consensus.cc:740] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.385814 10922 consensus_queue.cc:260] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.390242 10922 ts_tablet_manager.cc:1434] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260321 02:57:48.392640 10922 tablet_bootstrap.cc:492] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2: Bootstrap starting.
I20260321 02:57:48.393436 10733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e161a85964742b09837e0ae3dda87ca" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153c2de7193c4844b765382a198e9db2" is_pre_election: true
I20260321 02:57:48.394346 10733 raft_consensus.cc:2468] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in term 0.
I20260321 02:57:48.396309 10771 leader_election.cc:304] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters:
I20260321 02:57:48.384827 10923 tablet_bootstrap.cc:654] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.397902 10925 tablet_bootstrap.cc:492] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e: No bootstrap required, opened a new log
I20260321 02:57:48.398485 10925 ts_tablet_manager.cc:1403] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.009s
I20260321 02:57:48.400283 10930 raft_consensus.cc:2804] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:57:48.400713 10930 raft_consensus.cc:493] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:57:48.401074 10930 raft_consensus.cc:3060] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.405009 10925 raft_consensus.cc:359] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.406050 10925 raft_consensus.cc:385] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.406459 10925 raft_consensus.cc:740] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.407456 10925 consensus_queue.cc:260] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.413185 10930 raft_consensus.cc:515] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.414259 10925 ts_tablet_manager.cc:1434] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e: Time spent starting tablet: real 0.015s user 0.009s sys 0.000s
I20260321 02:57:48.415268 10925 tablet_bootstrap.cc:492] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e: Bootstrap starting.
I20260321 02:57:48.416955 10733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e161a85964742b09837e0ae3dda87ca" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153c2de7193c4844b765382a198e9db2"
I20260321 02:57:48.417768 10733 raft_consensus.cc:3060] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.418126 10922 tablet_bootstrap.cc:654] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2: Neither blocks nor log segments found. Creating new log.
W20260321 02:57:48.420557 10679 auto_rebalancer.cc:247] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260321 02:57:48.423009 10927 raft_consensus.cc:493] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:48.423249 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e161a85964742b09837e0ae3dda87ca" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed785fa1622433e8993e90b8c6c984e" is_pre_election: true
I20260321 02:57:48.423841 10927 raft_consensus.cc:515] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.426923 10927 leader_election.cc:290] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:48.424225 10882 raft_consensus.cc:2468] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in term 0.
I20260321 02:57:48.433352 10930 leader_election.cc:290] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 1 election: Requested vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:48.434697 10923 tablet_bootstrap.cc:492] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8: No bootstrap required, opened a new log
I20260321 02:57:48.435657 10925 tablet_bootstrap.cc:654] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.435851 10923 ts_tablet_manager.cc:1403] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8: Time spent bootstrapping tablet: real 0.065s user 0.024s sys 0.013s
I20260321 02:57:48.437680 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e161a85964742b09837e0ae3dda87ca" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
I20260321 02:57:48.438551 10882 raft_consensus.cc:3060] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.433528 10941 raft_consensus.cc:493] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:48.439659 10923 raft_consensus.cc:359] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.440457 10923 raft_consensus.cc:385] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.441232 10923 raft_consensus.cc:740] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.442229 10923 consensus_queue.cc:260] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.448948 10923 ts_tablet_manager.cc:1434] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8: Time spent starting tablet: real 0.012s user 0.008s sys 0.001s
I20260321 02:57:48.450490 10923 tablet_bootstrap.cc:492] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8: Bootstrap starting.
I20260321 02:57:48.454716 10941 raft_consensus.cc:515] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.455315 10882 raft_consensus.cc:2468] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in term 1.
I20260321 02:57:48.458667 10733 raft_consensus.cc:2468] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in term 1.
I20260321 02:57:48.458981 10771 leader_election.cc:304] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ed785fa1622433e8993e90b8c6c984e, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters:
I20260321 02:57:48.460475 10930 raft_consensus.cc:2804] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:48.464727 10923 tablet_bootstrap.cc:654] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.465018 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08f9bfdb4a464783b9b5b88025e96c4b" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed785fa1622433e8993e90b8c6c984e" is_pre_election: true
I20260321 02:57:48.465643 10732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08f9bfdb4a464783b9b5b88025e96c4b" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153c2de7193c4844b765382a198e9db2" is_pre_election: true
I20260321 02:57:48.466355 10732 raft_consensus.cc:2468] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in term 0.
W20260321 02:57:48.466857 10771 leader_election.cc:343] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:48.467768 10771 leader_election.cc:304] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters: 5ed785fa1622433e8993e90b8c6c984e
I20260321 02:57:48.462864 10941 leader_election.cc:290] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:48.471598 10941 raft_consensus.cc:2804] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:57:48.472079 10941 raft_consensus.cc:493] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:57:48.472450 10927 raft_consensus.cc:493] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:48.473150 10927 raft_consensus.cc:515] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.472494 10941 raft_consensus.cc:3060] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.476028 10927 leader_election.cc:290] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:48.483615 10930 raft_consensus.cc:697] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 LEADER]: Becoming Leader. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Running, Role: LEADER
I20260321 02:57:48.488533 10922 tablet_bootstrap.cc:492] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2: No bootstrap required, opened a new log
I20260321 02:57:48.489310 10922 ts_tablet_manager.cc:1403] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2: Time spent bootstrapping tablet: real 0.097s user 0.017s sys 0.025s
I20260321 02:57:48.488847 10930 consensus_queue.cc:237] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.496574 10922 raft_consensus.cc:359] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.497188 10922 raft_consensus.cc:385] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.497447 10922 raft_consensus.cc:740] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.498157 10922 consensus_queue.cc:260] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.505649 10922 ts_tablet_manager.cc:1434] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2: Time spent starting tablet: real 0.016s user 0.010s sys 0.000s
I20260321 02:57:48.506933 10922 tablet_bootstrap.cc:492] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2: Bootstrap starting.
I20260321 02:57:48.508621 10931 raft_consensus.cc:493] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:48.509271 10931 raft_consensus.cc:515] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.509971 10881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f35df046e02643dab408161cee9f00a4" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed785fa1622433e8993e90b8c6c984e" is_pre_election: true
W20260321 02:57:48.511884 10696 leader_election.cc:343] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:48.512796 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08f9bfdb4a464783b9b5b88025e96c4b" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed785fa1622433e8993e90b8c6c984e" is_pre_election: true
I20260321 02:57:48.513720 10922 tablet_bootstrap.cc:654] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.519556 10925 tablet_bootstrap.cc:492] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e: No bootstrap required, opened a new log
I20260321 02:57:48.520061 10925 ts_tablet_manager.cc:1403] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e: Time spent bootstrapping tablet: real 0.105s user 0.013s sys 0.019s
I20260321 02:57:48.521178 10941 raft_consensus.cc:515] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.525168 10931 leader_election.cc:290] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303)
I20260321 02:57:48.527870 10808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08f9bfdb4a464783b9b5b88025e96c4b" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" is_pre_election: true
I20260321 02:57:48.528997 10808 raft_consensus.cc:2393] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 153c2de7193c4844b765382a198e9db2 in current term 1: Already voted for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in this term.
I20260321 02:57:48.530406 10807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f35df046e02643dab408161cee9f00a4" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" is_pre_election: true
I20260321 02:57:48.531284 10807 raft_consensus.cc:2468] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 0.
I20260321 02:57:48.533578 10697 leader_election.cc:304] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters: 5ed785fa1622433e8993e90b8c6c984e
I20260321 02:57:48.534610 10927 raft_consensus.cc:2804] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:57:48.534952 10927 raft_consensus.cc:493] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:57:48.535277 10927 raft_consensus.cc:3060] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.535593 10941 leader_election.cc:290] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 1 election: Requested vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:48.537019 10925 raft_consensus.cc:359] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.537984 10925 raft_consensus.cc:385] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.538336 10925 raft_consensus.cc:740] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.539348 10922 tablet_bootstrap.cc:492] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2: No bootstrap required, opened a new log
I20260321 02:57:48.539104 10925 consensus_queue.cc:260] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.539686 10922 ts_tablet_manager.cc:1403] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.006s
I20260321 02:57:48.540778 10733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08f9bfdb4a464783b9b5b88025e96c4b" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153c2de7193c4844b765382a198e9db2"
I20260321 02:57:48.541369 10733 raft_consensus.cc:3060] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.541615 10925 ts_tablet_manager.cc:1434] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e: Time spent starting tablet: real 0.021s user 0.004s sys 0.002s
I20260321 02:57:48.542335 10922 raft_consensus.cc:359] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.543049 10922 raft_consensus.cc:385] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.543375 10922 raft_consensus.cc:740] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.544440 10922 consensus_queue.cc:260] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.549584 10733 raft_consensus.cc:2468] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in term 1.
I20260321 02:57:48.550825 10925 tablet_bootstrap.cc:492] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e: Bootstrap starting.
I20260321 02:57:48.551314 10771 leader_election.cc:304] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters:
I20260321 02:57:48.552561 10930 raft_consensus.cc:2804] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:48.553175 10930 raft_consensus.cc:697] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 LEADER]: Becoming Leader. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Running, Role: LEADER
I20260321 02:57:48.554306 10930 consensus_queue.cc:237] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.557804 10925 tablet_bootstrap.cc:654] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.560935 10923 tablet_bootstrap.cc:492] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8: No bootstrap required, opened a new log
I20260321 02:57:48.561761 10923 ts_tablet_manager.cc:1403] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8: Time spent bootstrapping tablet: real 0.112s user 0.017s sys 0.007s
I20260321 02:57:48.566087 10923 raft_consensus.cc:359] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.571254 10925 tablet_bootstrap.cc:492] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e: No bootstrap required, opened a new log
I20260321 02:57:48.582559 10925 ts_tablet_manager.cc:1403] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.002s
I20260321 02:57:48.581677 10631 catalog_manager.cc:5671] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2). New cstate: current_term: 1 leader_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:48.585559 10925 raft_consensus.cc:359] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.581681 10923 raft_consensus.cc:385] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.586287 10925 raft_consensus.cc:385] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.586393 10923 raft_consensus.cc:740] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.549662 10927 raft_consensus.cc:515] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.577713 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08f9bfdb4a464783b9b5b88025e96c4b" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
I20260321 02:57:48.587581 10923 consensus_queue.cc:260] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.588747 10922 ts_tablet_manager.cc:1434] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2: Time spent starting tablet: real 0.049s user 0.019s sys 0.031s
I20260321 02:57:48.589762 10922 tablet_bootstrap.cc:492] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2: Bootstrap starting.
I20260321 02:57:48.590008 10807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f35df046e02643dab408161cee9f00a4" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
I20260321 02:57:48.590796 10807 raft_consensus.cc:3060] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Advancing to term 1
W20260321 02:57:48.592249 10771 leader_election.cc:343] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:48.592540 10923 ts_tablet_manager.cc:1434] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8: Time spent starting tablet: real 0.030s user 0.008s sys 0.000s
I20260321 02:57:48.594866 10923 tablet_bootstrap.cc:492] T 082582ef8a7a483eaf98b4ced6214a57 P 9f26c8327bfe4a58ae8b67940c9f49f8: Bootstrap starting.
W20260321 02:57:48.576314 10696 leader_election.cc:343] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:48.601049 10696 leader_election.cc:304] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 153c2de7193c4844b765382a198e9db2; no voters: 5ed785fa1622433e8993e90b8c6c984e, 9f26c8327bfe4a58ae8b67940c9f49f8
I20260321 02:57:48.602011 10954 raft_consensus.cc:2749] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:57:48.596982 10733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2633115df76454eb4c74d33d2b0b8fe" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153c2de7193c4844b765382a198e9db2" is_pre_election: true
I20260321 02:57:48.586678 10925 raft_consensus.cc:740] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.603303 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f35df046e02643dab408161cee9f00a4" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
W20260321 02:57:48.604635 10696 leader_election.cc:343] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:48.604032 10925 consensus_queue.cc:260] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.599887 10922 tablet_bootstrap.cc:654] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2: Neither blocks nor log segments found. Creating new log.
W20260321 02:57:48.605571 10845 leader_election.cc:343] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:48.597843 10927 leader_election.cc:290] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 election: Requested vote from peers 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:48.607304 10923 tablet_bootstrap.cc:654] T 082582ef8a7a483eaf98b4ced6214a57 P 9f26c8327bfe4a58ae8b67940c9f49f8: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.609565 10808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2633115df76454eb4c74d33d2b0b8fe" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" is_pre_election: true
I20260321 02:57:48.610658 10925 ts_tablet_manager.cc:1434] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e: Time spent starting tablet: real 0.028s user 0.003s sys 0.003s
W20260321 02:57:48.611624 10846 leader_election.cc:343] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:48.612533 10925 tablet_bootstrap.cc:492] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e: Bootstrap starting.
I20260321 02:57:48.602257 10807 raft_consensus.cc:2468] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 1.
I20260321 02:57:48.614459 10697 leader_election.cc:304] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters: 5ed785fa1622433e8993e90b8c6c984e
I20260321 02:57:48.616168 10927 raft_consensus.cc:2804] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:48.616823 10927 raft_consensus.cc:697] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Running, Role: LEADER
I20260321 02:57:48.617889 10927 consensus_queue.cc:237] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.612293 10846 leader_election.cc:304] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ed785fa1622433e8993e90b8c6c984e; no voters: 153c2de7193c4844b765382a198e9db2, 9f26c8327bfe4a58ae8b67940c9f49f8
I20260321 02:57:48.626626 10925 tablet_bootstrap.cc:654] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.628508 10931 raft_consensus.cc:2749] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:57:48.631656 10923 tablet_bootstrap.cc:492] T 082582ef8a7a483eaf98b4ced6214a57 P 9f26c8327bfe4a58ae8b67940c9f49f8: No bootstrap required, opened a new log
I20260321 02:57:48.632241 10923 ts_tablet_manager.cc:1403] T 082582ef8a7a483eaf98b4ced6214a57 P 9f26c8327bfe4a58ae8b67940c9f49f8: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.020s
I20260321 02:57:48.636749 10922 tablet_bootstrap.cc:492] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2: No bootstrap required, opened a new log
I20260321 02:57:48.637478 10922 ts_tablet_manager.cc:1403] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2: Time spent bootstrapping tablet: real 0.048s user 0.021s sys 0.008s
I20260321 02:57:48.636957 10923 raft_consensus.cc:359] T 082582ef8a7a483eaf98b4ced6214a57 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.638037 10923 raft_consensus.cc:385] T 082582ef8a7a483eaf98b4ced6214a57 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.638432 10954 raft_consensus.cc:493] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:48.638669 10923 raft_consensus.cc:740] T 082582ef8a7a483eaf98b4ced6214a57 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.638911 10954 raft_consensus.cc:515] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.639442 10923 consensus_queue.cc:260] T 082582ef8a7a483eaf98b4ced6214a57 P 9f26c8327bfe4a58ae8b67940c9f49f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.641220 10954 leader_election.cc:290] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:48.641083 10922 raft_consensus.cc:359] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.642292 10923 ts_tablet_manager.cc:1434] T 082582ef8a7a483eaf98b4ced6214a57 P 9f26c8327bfe4a58ae8b67940c9f49f8: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260321 02:57:48.642235 10922 raft_consensus.cc:385] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.642901 10922 raft_consensus.cc:740] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.643471 10923 tablet_bootstrap.cc:492] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8: Bootstrap starting.
I20260321 02:57:48.643464 10807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "082582ef8a7a483eaf98b4ced6214a57" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" is_pre_election: true
I20260321 02:57:48.644222 10807 raft_consensus.cc:2468] T 082582ef8a7a483eaf98b4ced6214a57 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 0.
I20260321 02:57:48.644062 10922 consensus_queue.cc:260] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.645766 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "082582ef8a7a483eaf98b4ced6214a57" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed785fa1622433e8993e90b8c6c984e" is_pre_election: true
I20260321 02:57:48.646698 10930 raft_consensus.cc:493] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:48.647262 10922 ts_tablet_manager.cc:1434] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260321 02:57:48.647257 10930 raft_consensus.cc:515] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.648661 10922 tablet_bootstrap.cc:492] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 153c2de7193c4844b765382a198e9db2: Bootstrap starting.
I20260321 02:57:48.649608 10930 leader_election.cc:290] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
W20260321 02:57:48.650825 10696 leader_election.cc:343] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:48.651224 10923 tablet_bootstrap.cc:654] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.652959 10697 leader_election.cc:304] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters: 5ed785fa1622433e8993e90b8c6c984e
I20260321 02:57:48.653105 10733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8619a3b4cc1b4d59b5a312ac0bc01017" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153c2de7193c4844b765382a198e9db2" is_pre_election: true
I20260321 02:57:48.654065 10931 raft_consensus.cc:493] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:48.654526 10931 raft_consensus.cc:515] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.654999 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8619a3b4cc1b4d59b5a312ac0bc01017" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed785fa1622433e8993e90b8c6c984e" is_pre_election: true
I20260321 02:57:48.655584 10882 raft_consensus.cc:2468] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in term 0.
I20260321 02:57:48.656163 10922 tablet_bootstrap.cc:654] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 153c2de7193c4844b765382a198e9db2: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.656534 10931 leader_election.cc:290] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303)
I20260321 02:57:48.658932 10954 raft_consensus.cc:2804] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:57:48.659353 10954 raft_consensus.cc:493] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:57:48.659709 10954 raft_consensus.cc:3060] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.663090 10771 leader_election.cc:304] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ed785fa1622433e8993e90b8c6c984e, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters:
I20260321 02:57:48.663928 10930 raft_consensus.cc:2804] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260321 02:57:48.664083 10771 leader_election.cc:343] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:48.664355 10930 raft_consensus.cc:493] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:57:48.664721 10930 raft_consensus.cc:3060] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.665215 10807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8619a3b4cc1b4d59b5a312ac0bc01017" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" is_pre_election: true
I20260321 02:57:48.674245 10627 catalog_manager.cc:5671] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2). New cstate: current_term: 1 leader_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:48.678236 10922 tablet_bootstrap.cc:492] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 153c2de7193c4844b765382a198e9db2: No bootstrap required, opened a new log
I20260321 02:57:48.678817 10922 ts_tablet_manager.cc:1403] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 153c2de7193c4844b765382a198e9db2: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.008s
I20260321 02:57:48.679041 10954 raft_consensus.cc:515] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.681434 10922 raft_consensus.cc:359] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.683023 10733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8619a3b4cc1b4d59b5a312ac0bc01017" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153c2de7193c4844b765382a198e9db2" is_pre_election: true
I20260321 02:57:48.690193 10922 raft_consensus.cc:385] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.688853 10930 raft_consensus.cc:515] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.690537 10922 raft_consensus.cc:740] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.688030 10629 catalog_manager.cc:5671] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 153c2de7193c4844b765382a198e9db2 (127.9.213.1). New cstate: current_term: 1 leader_uuid: "153c2de7193c4844b765382a198e9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:48.691419 10807 raft_consensus.cc:2393] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ed785fa1622433e8993e90b8c6c984e in current term 1: Already voted for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in this term.
I20260321 02:57:48.691473 10922 consensus_queue.cc:260] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 153c2de7193c4844b765382a198e9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.692525 10733 raft_consensus.cc:2468] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ed785fa1622433e8993e90b8c6c984e in term 0.
I20260321 02:57:48.694094 10922 ts_tablet_manager.cc:1434] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 153c2de7193c4844b765382a198e9db2: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260321 02:57:48.695056 10922 tablet_bootstrap.cc:492] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2: Bootstrap starting.
I20260321 02:57:48.696341 10845 leader_election.cc:304] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 5ed785fa1622433e8993e90b8c6c984e; no voters: 9f26c8327bfe4a58ae8b67940c9f49f8
I20260321 02:57:48.697170 10931 raft_consensus.cc:2804] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:57:48.689857 10925 tablet_bootstrap.cc:492] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e: No bootstrap required, opened a new log
I20260321 02:57:48.697546 10931 raft_consensus.cc:493] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:57:48.697928 10931 raft_consensus.cc:3060] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.698063 10925 ts_tablet_manager.cc:1403] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e: Time spent bootstrapping tablet: real 0.086s user 0.017s sys 0.003s
I20260321 02:57:48.702207 10807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "082582ef8a7a483eaf98b4ced6214a57" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
I20260321 02:57:48.702513 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "082582ef8a7a483eaf98b4ced6214a57" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
I20260321 02:57:48.702807 10807 raft_consensus.cc:3060] T 082582ef8a7a483eaf98b4ced6214a57 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.703197 10954 leader_election.cc:290] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 election: Requested vote from peers 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
W20260321 02:57:48.704157 10696 leader_election.cc:343] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:48.704684 10922 tablet_bootstrap.cc:654] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.705111 10925 raft_consensus.cc:359] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.706012 10925 raft_consensus.cc:385] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.706379 10925 raft_consensus.cc:740] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.708986 10930 leader_election.cc:290] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 1 election: Requested vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:48.710534 10733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8619a3b4cc1b4d59b5a312ac0bc01017" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153c2de7193c4844b765382a198e9db2"
I20260321 02:57:48.711099 10733 raft_consensus.cc:3060] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.711876 10923 tablet_bootstrap.cc:492] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8: No bootstrap required, opened a new log
I20260321 02:57:48.712106 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8619a3b4cc1b4d59b5a312ac0bc01017" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
I20260321 02:57:48.712402 10923 ts_tablet_manager.cc:1403] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8: Time spent bootstrapping tablet: real 0.069s user 0.028s sys 0.011s
I20260321 02:57:48.715308 10923 raft_consensus.cc:359] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.716068 10923 raft_consensus.cc:385] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.716356 10923 raft_consensus.cc:740] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.717275 10923 consensus_queue.cc:260] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.707464 10925 consensus_queue.cc:260] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.720422 10923 ts_tablet_manager.cc:1434] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260321 02:57:48.721453 10923 tablet_bootstrap.cc:492] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8: Bootstrap starting.
I20260321 02:57:48.722537 10931 raft_consensus.cc:515] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.723742 10882 raft_consensus.cc:2393] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in current term 1: Already voted for candidate 5ed785fa1622433e8993e90b8c6c984e in this term.
I20260321 02:57:48.727836 10923 tablet_bootstrap.cc:654] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.727820 10807 raft_consensus.cc:2468] T 082582ef8a7a483eaf98b4ced6214a57 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 1.
I20260321 02:57:48.729107 10697 leader_election.cc:304] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters: 5ed785fa1622433e8993e90b8c6c984e
I20260321 02:57:48.729538 10733 raft_consensus.cc:2468] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in term 1.
I20260321 02:57:48.730960 10954 raft_consensus.cc:2804] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:48.731015 10771 leader_election.cc:304] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters: 5ed785fa1622433e8993e90b8c6c984e
I20260321 02:57:48.731623 10954 raft_consensus.cc:697] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Running, Role: LEADER
I20260321 02:57:48.732578 10954 consensus_queue.cc:237] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.734258 10733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8619a3b4cc1b4d59b5a312ac0bc01017" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153c2de7193c4844b765382a198e9db2"
I20260321 02:57:48.736027 10930 raft_consensus.cc:2804] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:48.741225 10930 raft_consensus.cc:697] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 LEADER]: Becoming Leader. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Running, Role: LEADER
I20260321 02:57:48.742028 10930 consensus_queue.cc:237] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.745543 10922 tablet_bootstrap.cc:492] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2: No bootstrap required, opened a new log
I20260321 02:57:48.746122 10922 ts_tablet_manager.cc:1403] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2: Time spent bootstrapping tablet: real 0.051s user 0.027s sys 0.015s
I20260321 02:57:48.735764 10733 raft_consensus.cc:2393] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5ed785fa1622433e8993e90b8c6c984e in current term 1: Already voted for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in this term.
I20260321 02:57:48.749402 10931 leader_election.cc:290] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 election: Requested vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303)
I20260321 02:57:48.749577 10922 raft_consensus.cc:359] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.750414 10922 raft_consensus.cc:385] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.750741 10922 raft_consensus.cc:740] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.751740 10922 consensus_queue.cc:260] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.754763 10922 ts_tablet_manager.cc:1434] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260321 02:57:48.755209 10931 raft_consensus.cc:493] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:48.755738 10931 raft_consensus.cc:515] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.759402 10922 tablet_bootstrap.cc:492] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2: Bootstrap starting.
I20260321 02:57:48.757799 10629 catalog_manager.cc:5671] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2). New cstate: current_term: 1 leader_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:48.760622 10807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8619a3b4cc1b4d59b5a312ac0bc01017" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
I20260321 02:57:48.770699 10931 leader_election.cc:290] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303)
I20260321 02:57:48.771932 10807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d08251e74b419599f24a910bc9e538" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" is_pre_election: true
W20260321 02:57:48.773445 10846 leader_election.cc:343] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:48.773803 10845 leader_election.cc:304] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ed785fa1622433e8993e90b8c6c984e; no voters: 153c2de7193c4844b765382a198e9db2, 9f26c8327bfe4a58ae8b67940c9f49f8
I20260321 02:57:48.774170 10923 tablet_bootstrap.cc:492] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8: No bootstrap required, opened a new log
I20260321 02:57:48.774736 10733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d08251e74b419599f24a910bc9e538" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153c2de7193c4844b765382a198e9db2" is_pre_election: true
I20260321 02:57:48.775537 10733 raft_consensus.cc:2468] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ed785fa1622433e8993e90b8c6c984e in term 0.
I20260321 02:57:48.774763 10923 ts_tablet_manager.cc:1403] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.020s
I20260321 02:57:48.775920 10970 raft_consensus.cc:2749] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260321 02:57:48.775944 10922 tablet_bootstrap.cc:654] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.777456 10925 ts_tablet_manager.cc:1434] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e: Time spent starting tablet: real 0.079s user 0.014s sys 0.012s
I20260321 02:57:48.778857 10923 raft_consensus.cc:359] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.779652 10923 raft_consensus.cc:385] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.779913 10923 raft_consensus.cc:740] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.780684 10923 consensus_queue.cc:260] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.783197 10923 ts_tablet_manager.cc:1434] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260321 02:57:48.784162 10923 tablet_bootstrap.cc:492] T 60d9453110c241f4aaa3740c3a43e6b6 P 9f26c8327bfe4a58ae8b67940c9f49f8: Bootstrap starting.
I20260321 02:57:48.785813 10845 leader_election.cc:304] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 5ed785fa1622433e8993e90b8c6c984e; no voters: 9f26c8327bfe4a58ae8b67940c9f49f8
I20260321 02:57:48.786974 10970 raft_consensus.cc:2804] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:57:48.787478 10970 raft_consensus.cc:493] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:57:48.787890 10970 raft_consensus.cc:3060] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.788304 10969 raft_consensus.cc:493] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:48.788816 10969 raft_consensus.cc:515] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.790704 10954 raft_consensus.cc:493] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:48.791026 10969 leader_election.cc:290] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303)
I20260321 02:57:48.791208 10954 raft_consensus.cc:515] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.791994 10925 tablet_bootstrap.cc:492] T 08f9bfdb4a464783b9b5b88025e96c4b P 5ed785fa1622433e8993e90b8c6c984e: Bootstrap starting.
I20260321 02:57:48.789534 10631 catalog_manager.cc:5671] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 153c2de7193c4844b765382a198e9db2 (127.9.213.1). New cstate: current_term: 1 leader_uuid: "153c2de7193c4844b765382a198e9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:48.796867 10954 leader_election.cc:290] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:48.796996 10808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56d5d5429b743bc8d2cb6fcb999f86e" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" is_pre_election: true
I20260321 02:57:48.797924 10808 raft_consensus.cc:2468] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ed785fa1622433e8993e90b8c6c984e in term 0.
I20260321 02:57:48.799047 10846 leader_election.cc:304] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ed785fa1622433e8993e90b8c6c984e, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters:
I20260321 02:57:48.799921 10969 raft_consensus.cc:2804] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:57:48.800338 10969 raft_consensus.cc:493] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:57:48.800504 10923 tablet_bootstrap.cc:654] T 60d9453110c241f4aaa3740c3a43e6b6 P 9f26c8327bfe4a58ae8b67940c9f49f8: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.800669 10969 raft_consensus.cc:3060] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.801661 10922 tablet_bootstrap.cc:492] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2: No bootstrap required, opened a new log
I20260321 02:57:48.803709 10925 tablet_bootstrap.cc:654] T 08f9bfdb4a464783b9b5b88025e96c4b P 5ed785fa1622433e8993e90b8c6c984e: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.804328 10922 ts_tablet_manager.cc:1403] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2: Time spent bootstrapping tablet: real 0.049s user 0.017s sys 0.016s
I20260321 02:57:48.807216 10922 raft_consensus.cc:359] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.808136 10922 raft_consensus.cc:385] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.807998 10969 raft_consensus.cc:515] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.808558 10922 raft_consensus.cc:740] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.809461 10922 consensus_queue.cc:260] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.811162 10808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56d5d5429b743bc8d2cb6fcb999f86e" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
I20260321 02:57:48.811738 10808 raft_consensus.cc:3060] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.816385 10922 ts_tablet_manager.cc:1434] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2: Time spent starting tablet: real 0.012s user 0.007s sys 0.001s
I20260321 02:57:48.819209 10808 raft_consensus.cc:2468] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed785fa1622433e8993e90b8c6c984e in term 1.
I20260321 02:57:48.819808 10969 leader_election.cc:290] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 election: Requested vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303)
I20260321 02:57:48.821045 10846 leader_election.cc:304] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ed785fa1622433e8993e90b8c6c984e, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters:
I20260321 02:57:48.821859 10969 raft_consensus.cc:2804] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:48.822043 10954 raft_consensus.cc:493] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:48.822681 10954 raft_consensus.cc:515] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.823210 10969 raft_consensus.cc:697] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 1 LEADER]: Becoming Leader. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Running, Role: LEADER
I20260321 02:57:48.824370 10807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2633115df76454eb4c74d33d2b0b8fe" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" is_pre_election: true
I20260321 02:57:48.825409 10954 leader_election.cc:290] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
W20260321 02:57:48.827210 10697 leader_election.cc:343] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:48.827589 10807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a1c86bd318d452e9446f7ab8783c20f" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" is_pre_election: true
I20260321 02:57:48.828258 10807 raft_consensus.cc:2468] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 0.
I20260321 02:57:48.829320 10697 leader_election.cc:304] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters:
I20260321 02:57:48.830196 10954 raft_consensus.cc:2804] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:57:48.830552 10954 raft_consensus.cc:493] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:57:48.830888 10954 raft_consensus.cc:3060] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.837883 10954 raft_consensus.cc:515] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.839073 10925 tablet_bootstrap.cc:492] T 08f9bfdb4a464783b9b5b88025e96c4b P 5ed785fa1622433e8993e90b8c6c984e: No bootstrap required, opened a new log
I20260321 02:57:48.839762 10925 ts_tablet_manager.cc:1403] T 08f9bfdb4a464783b9b5b88025e96c4b P 5ed785fa1622433e8993e90b8c6c984e: Time spent bootstrapping tablet: real 0.048s user 0.019s sys 0.019s
W20260321 02:57:48.840382 10845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:42123, user_credentials={real_user=slave}} blocked reactor thread for 54673.9us
I20260321 02:57:48.840592 10807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a1c86bd318d452e9446f7ab8783c20f" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
I20260321 02:57:48.841362 10807 raft_consensus.cc:3060] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.842131 10733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56d5d5429b743bc8d2cb6fcb999f86e" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153c2de7193c4844b765382a198e9db2" is_pre_election: true
I20260321 02:57:48.842769 10733 raft_consensus.cc:2468] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5ed785fa1622433e8993e90b8c6c984e in term 0.
I20260321 02:57:48.843704 10925 raft_consensus.cc:359] T 08f9bfdb4a464783b9b5b88025e96c4b P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.844959 10925 raft_consensus.cc:385] T 08f9bfdb4a464783b9b5b88025e96c4b P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.845364 10925 raft_consensus.cc:740] T 08f9bfdb4a464783b9b5b88025e96c4b P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.844069 10969 consensus_queue.cc:237] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.847759 10925 consensus_queue.cc:260] T 08f9bfdb4a464783b9b5b88025e96c4b P 5ed785fa1622433e8993e90b8c6c984e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.849259 10807 raft_consensus.cc:2468] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 1.
I20260321 02:57:48.849198 10733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56d5d5429b743bc8d2cb6fcb999f86e" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153c2de7193c4844b765382a198e9db2"
I20260321 02:57:48.849853 10954 leader_election.cc:290] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 election: Requested vote from peers 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:48.849941 10733 raft_consensus.cc:3060] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.852421 10697 leader_election.cc:304] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters:
I20260321 02:57:48.853374 10964 raft_consensus.cc:2804] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:48.853466 10931 raft_consensus.cc:493] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:48.853879 10964 raft_consensus.cc:697] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Running, Role: LEADER
I20260321 02:57:48.853921 10931 raft_consensus.cc:515] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.854233 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2633115df76454eb4c74d33d2b0b8fe" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed785fa1622433e8993e90b8c6c984e" is_pre_election: true
I20260321 02:57:48.854514 10881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a1c86bd318d452e9446f7ab8783c20f" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed785fa1622433e8993e90b8c6c984e" is_pre_election: true
I20260321 02:57:48.854847 10882 raft_consensus.cc:2468] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 0.
I20260321 02:57:48.854794 10964 consensus_queue.cc:237] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.855886 10931 leader_election.cc:290] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303)
I20260321 02:57:48.856530 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a1c86bd318d452e9446f7ab8783c20f" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
I20260321 02:57:48.856967 10925 ts_tablet_manager.cc:1434] T 08f9bfdb4a464783b9b5b88025e96c4b P 5ed785fa1622433e8993e90b8c6c984e: Time spent starting tablet: real 0.017s user 0.008s sys 0.001s
W20260321 02:57:48.857586 10696 leader_election.cc:343] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:48.857985 10925 tablet_bootstrap.cc:492] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e: Bootstrap starting.
I20260321 02:57:48.858536 10696 leader_election.cc:304] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 5ed785fa1622433e8993e90b8c6c984e; no voters: 9f26c8327bfe4a58ae8b67940c9f49f8
I20260321 02:57:48.858755 10732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "082582ef8a7a483eaf98b4ced6214a57" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153c2de7193c4844b765382a198e9db2" is_pre_election: true
I20260321 02:57:48.860117 10927 raft_consensus.cc:2804] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:57:48.860498 10927 raft_consensus.cc:493] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:57:48.860844 10927 raft_consensus.cc:3060] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.861218 10970 raft_consensus.cc:515] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.863361 10970 leader_election.cc:290] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 election: Requested vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303)
W20260321 02:57:48.864545 10696 leader_election.cc:343] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:48.864820 10732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d08251e74b419599f24a910bc9e538" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153c2de7193c4844b765382a198e9db2"
I20260321 02:57:48.865584 10732 raft_consensus.cc:3060] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.867895 10808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d08251e74b419599f24a910bc9e538" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
I20260321 02:57:48.868647 10808 raft_consensus.cc:3060] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.868726 10923 tablet_bootstrap.cc:492] T 60d9453110c241f4aaa3740c3a43e6b6 P 9f26c8327bfe4a58ae8b67940c9f49f8: No bootstrap required, opened a new log
I20260321 02:57:48.869726 10923 ts_tablet_manager.cc:1403] T 60d9453110c241f4aaa3740c3a43e6b6 P 9f26c8327bfe4a58ae8b67940c9f49f8: Time spent bootstrapping tablet: real 0.086s user 0.018s sys 0.029s
I20260321 02:57:48.872330 10923 raft_consensus.cc:359] T 60d9453110c241f4aaa3740c3a43e6b6 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.872982 10923 raft_consensus.cc:385] T 60d9453110c241f4aaa3740c3a43e6b6 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.873246 10923 raft_consensus.cc:740] T 60d9453110c241f4aaa3740c3a43e6b6 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.873570 10732 raft_consensus.cc:2468] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed785fa1622433e8993e90b8c6c984e in term 1.
I20260321 02:57:48.874611 10733 raft_consensus.cc:2468] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed785fa1622433e8993e90b8c6c984e in term 1.
I20260321 02:57:48.875020 10845 leader_election.cc:304] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 5ed785fa1622433e8993e90b8c6c984e; no voters:
I20260321 02:57:48.876591 10970 raft_consensus.cc:2804] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:48.877171 10970 raft_consensus.cc:697] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 1 LEADER]: Becoming Leader. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Running, Role: LEADER
I20260321 02:57:48.878126 10970 consensus_queue.cc:237] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.879297 10807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "082582ef8a7a483eaf98b4ced6214a57" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" is_pre_election: true
I20260321 02:57:48.878249 10923 consensus_queue.cc:260] T 60d9453110c241f4aaa3740c3a43e6b6 P 9f26c8327bfe4a58ae8b67940c9f49f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.880093 10807 raft_consensus.cc:2393] T 082582ef8a7a483eaf98b4ced6214a57 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ed785fa1622433e8993e90b8c6c984e in current term 1: Already voted for candidate 153c2de7193c4844b765382a198e9db2 in this term.
I20260321 02:57:48.882025 10923 ts_tablet_manager.cc:1434] T 60d9453110c241f4aaa3740c3a43e6b6 P 9f26c8327bfe4a58ae8b67940c9f49f8: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260321 02:57:48.881319 10629 catalog_manager.cc:5671] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3). New cstate: current_term: 1 leader_uuid: "5ed785fa1622433e8993e90b8c6c984e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: HEALTHY } } }
I20260321 02:57:48.882983 10923 tablet_bootstrap.cc:492] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8: Bootstrap starting.
I20260321 02:57:48.878383 10927 raft_consensus.cc:515] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.887490 10806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2633115df76454eb4c74d33d2b0b8fe" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
W20260321 02:57:48.889513 10697 leader_election.cc:343] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303): Illegal state: must be running to vote when last-logged opid is not known
W20260321 02:57:48.890594 10759 tablet.cc:2379] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260321 02:57:48.897853 10923 tablet_bootstrap.cc:654] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.899011 10846 leader_election.cc:304] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ed785fa1622433e8993e90b8c6c984e; no voters: 153c2de7193c4844b765382a198e9db2, 9f26c8327bfe4a58ae8b67940c9f49f8
I20260321 02:57:48.899852 10925 tablet_bootstrap.cc:654] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.900501 10927 leader_election.cc:290] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 election: Requested vote from peers 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:48.901422 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2633115df76454eb4c74d33d2b0b8fe" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
I20260321 02:57:48.901980 10882 raft_consensus.cc:3060] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.907816 10969 raft_consensus.cc:3060] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.910009 10923 tablet_bootstrap.cc:492] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8: No bootstrap required, opened a new log
I20260321 02:57:48.910670 10923 ts_tablet_manager.cc:1403] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.003s
I20260321 02:57:48.915845 10969 raft_consensus.cc:2749] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:57:48.916801 10923 raft_consensus.cc:359] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.919535 10923 raft_consensus.cc:385] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.918606 10629 catalog_manager.cc:5671] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 153c2de7193c4844b765382a198e9db2 (127.9.213.1). New cstate: current_term: 1 leader_uuid: "153c2de7193c4844b765382a198e9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:48.917905 10882 raft_consensus.cc:2468] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 1.
I20260321 02:57:48.920673 10925 tablet_bootstrap.cc:492] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e: No bootstrap required, opened a new log
I20260321 02:57:48.920970 10696 leader_election.cc:304] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 5ed785fa1622433e8993e90b8c6c984e; no voters: 9f26c8327bfe4a58ae8b67940c9f49f8
I20260321 02:57:48.921257 10925 ts_tablet_manager.cc:1403] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e: Time spent bootstrapping tablet: real 0.063s user 0.019s sys 0.009s
I20260321 02:57:48.922179 10927 raft_consensus.cc:2804] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:48.919816 10923 raft_consensus.cc:740] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.923007 10927 raft_consensus.cc:697] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Running, Role: LEADER
I20260321 02:57:48.924386 10808 raft_consensus.cc:2468] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed785fa1622433e8993e90b8c6c984e in term 1.
I20260321 02:57:48.924620 10925 raft_consensus.cc:359] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.925558 10925 raft_consensus.cc:385] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.923070 10923 consensus_queue.cc:260] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.929064 10923 ts_tablet_manager.cc:1434] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8: Time spent starting tablet: real 0.018s user 0.010s sys 0.000s
I20260321 02:57:48.930696 10964 raft_consensus.cc:493] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:48.925937 10925 raft_consensus.cc:740] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.923942 10927 consensus_queue.cc:237] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.931218 10964 raft_consensus.cc:515] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.933418 10964 leader_election.cc:290] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:48.933782 10930 raft_consensus.cc:493] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:48.934271 10930 raft_consensus.cc:515] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.935247 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60d9453110c241f4aaa3740c3a43e6b6" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed785fa1622433e8993e90b8c6c984e" is_pre_election: true
I20260321 02:57:48.936211 10930 leader_election.cc:290] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:48.935557 10631 catalog_manager.cc:5671] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3). New cstate: current_term: 1 leader_uuid: "5ed785fa1622433e8993e90b8c6c984e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: HEALTHY } } }
I20260321 02:57:48.936506 10808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60d9453110c241f4aaa3740c3a43e6b6" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" is_pre_election: true
I20260321 02:57:48.937290 10808 raft_consensus.cc:2468] T 60d9453110c241f4aaa3740c3a43e6b6 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 0.
I20260321 02:57:48.937570 10925 consensus_queue.cc:260] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.938180 10733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2633115df76454eb4c74d33d2b0b8fe" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153c2de7193c4844b765382a198e9db2" is_pre_election: true
W20260321 02:57:48.939194 10696 leader_election.cc:343] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:57:48.939823 10925 ts_tablet_manager.cc:1434] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260321 02:57:48.940205 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2633115df76454eb4c74d33d2b0b8fe" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed785fa1622433e8993e90b8c6c984e" is_pre_election: true
I20260321 02:57:48.940415 10697 leader_election.cc:304] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters: 5ed785fa1622433e8993e90b8c6c984e
I20260321 02:57:48.940799 10925 tablet_bootstrap.cc:492] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e: Bootstrap starting.
I20260321 02:57:48.940994 10882 raft_consensus.cc:2393] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in current term 1: Already voted for candidate 153c2de7193c4844b765382a198e9db2 in this term.
I20260321 02:57:48.941596 10964 raft_consensus.cc:2804] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:57:48.941985 10964 raft_consensus.cc:493] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:57:48.942363 10964 raft_consensus.cc:3060] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.942440 10771 leader_election.cc:304] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f26c8327bfe4a58ae8b67940c9f49f8; no voters: 153c2de7193c4844b765382a198e9db2, 5ed785fa1622433e8993e90b8c6c984e
I20260321 02:57:48.943758 10930 raft_consensus.cc:3060] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.948050 10925 tablet_bootstrap.cc:654] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.950393 10964 raft_consensus.cc:515] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.955580 10930 raft_consensus.cc:2749] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:57:48.955786 10925 tablet_bootstrap.cc:492] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e: No bootstrap required, opened a new log
I20260321 02:57:48.956713 10925 ts_tablet_manager.cc:1403] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.004s
I20260321 02:57:48.959220 10925 raft_consensus.cc:359] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.959630 10964 leader_election.cc:290] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 election: Requested vote from peers 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:48.959859 10925 raft_consensus.cc:385] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.960178 10925 raft_consensus.cc:740] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.960870 10925 consensus_queue.cc:260] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.963640 10808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60d9453110c241f4aaa3740c3a43e6b6" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
I20260321 02:57:48.964315 10808 raft_consensus.cc:3060] T 60d9453110c241f4aaa3740c3a43e6b6 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.965052 10925 ts_tablet_manager.cc:1434] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260321 02:57:48.964628 10627 catalog_manager.cc:5671] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 153c2de7193c4844b765382a198e9db2 (127.9.213.1). New cstate: current_term: 1 leader_uuid: "153c2de7193c4844b765382a198e9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:48.966030 10925 tablet_bootstrap.cc:492] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e: Bootstrap starting.
I20260321 02:57:48.968571 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60d9453110c241f4aaa3740c3a43e6b6" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
I20260321 02:57:48.969193 10882 raft_consensus.cc:3060] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:48.971480 10925 tablet_bootstrap.cc:654] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e: Neither blocks nor log segments found. Creating new log.
I20260321 02:57:48.975924 10808 raft_consensus.cc:2468] T 60d9453110c241f4aaa3740c3a43e6b6 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 1.
I20260321 02:57:48.977370 10930 consensus_queue.cc:1048] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:57:48.983006 10882 raft_consensus.cc:2468] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 1.
I20260321 02:57:48.984532 10696 leader_election.cc:304] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 5ed785fa1622433e8993e90b8c6c984e; no voters:
I20260321 02:57:48.985795 10964 raft_consensus.cc:2804] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:57:48.986538 10964 raft_consensus.cc:697] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Becoming Leader. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Running, Role: LEADER
I20260321 02:57:48.987396 10964 consensus_queue.cc:237] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.991281 10925 tablet_bootstrap.cc:492] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e: No bootstrap required, opened a new log
I20260321 02:57:48.991791 10925 ts_tablet_manager.cc:1403] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.012s
I20260321 02:57:48.994439 10925 raft_consensus.cc:359] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.995138 10925 raft_consensus.cc:385] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:57:48.994781 10930 consensus_queue.cc:1048] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:57:48.995441 10925 raft_consensus.cc:740] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Initialized, Role: FOLLOWER
I20260321 02:57:48.996311 10925 consensus_queue.cc:260] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:48.998795 10925 ts_tablet_manager.cc:1434] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20260321 02:57:49.004853 10627 catalog_manager.cc:5671] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 153c2de7193c4844b765382a198e9db2 (127.9.213.1). New cstate: current_term: 1 leader_uuid: "153c2de7193c4844b765382a198e9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:49.031737 10881 raft_consensus.cc:3060] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:49.037528 10964 consensus_queue.cc:1048] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:49.047986 10927 consensus_queue.cc:1048] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:49.084453 10881 raft_consensus.cc:3060] T 08f9bfdb4a464783b9b5b88025e96c4b P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:49.090678 10948 consensus_queue.cc:1048] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:49.101990 10994 consensus_queue.cc:1048] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:57:49.175130 10991 consensus_queue.cc:1048] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:57:49.186347 10964 consensus_queue.cc:1048] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:57:49.190043 10994 consensus_queue.cc:1048] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:49.205482 10994 consensus_queue.cc:1048] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:49.228446 10808 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6e161a85964742b09837e0ae3dda87ca"
dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
mode: GRACEFUL
new_leader_uuid: "153c2de7193c4844b765382a198e9db2"
from {username='slave'} at 127.0.0.1:37816
I20260321 02:57:49.228986 10808 raft_consensus.cc:606] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 LEADER]: Received request to transfer leadership to TS 153c2de7193c4844b765382a198e9db2
I20260321 02:57:49.236562 10808 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "08f9bfdb4a464783b9b5b88025e96c4b"
dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
mode: GRACEFUL
new_leader_uuid: "153c2de7193c4844b765382a198e9db2"
from {username='slave'} at 127.0.0.1:37816
I20260321 02:57:49.237032 10808 raft_consensus.cc:606] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 LEADER]: Received request to transfer leadership to TS 153c2de7193c4844b765382a198e9db2
I20260321 02:57:49.254940 10881 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a56d5d5429b743bc8d2cb6fcb999f86e"
dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
mode: GRACEFUL
new_leader_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
from {username='slave'} at 127.0.0.1:34908
I20260321 02:57:49.255534 10881 raft_consensus.cc:606] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 1 LEADER]: Received request to transfer leadership to TS 9f26c8327bfe4a58ae8b67940c9f49f8
I20260321 02:57:49.260968 10881 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e1d08251e74b419599f24a910bc9e538"
dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
mode: GRACEFUL
new_leader_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
from {username='slave'} at 127.0.0.1:34908
I20260321 02:57:49.261586 10881 raft_consensus.cc:606] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 1 LEADER]: Received request to transfer leadership to TS 9f26c8327bfe4a58ae8b67940c9f49f8
I20260321 02:57:49.275693 10970 raft_consensus.cc:493] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:57:49.276099 10733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b2633115df76454eb4c74d33d2b0b8fe"
dest_uuid: "153c2de7193c4844b765382a198e9db2"
mode: GRACEFUL
new_leader_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
from {username='slave'} at 127.0.0.1:59914
I20260321 02:57:49.276248 10970 raft_consensus.cc:515] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:49.276718 10733 raft_consensus.cc:606] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Received request to transfer leadership to TS 9f26c8327bfe4a58ae8b67940c9f49f8
I20260321 02:57:49.277604 10969 consensus_queue.cc:1048] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:49.278260 10970 leader_election.cc:290] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303)
I20260321 02:57:49.279320 10733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a1c86bd318d452e9446f7ab8783c20f" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153c2de7193c4844b765382a198e9db2" is_pre_election: true
I20260321 02:57:49.280035 10808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a1c86bd318d452e9446f7ab8783c20f" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" is_pre_election: true
I20260321 02:57:49.280798 10808 raft_consensus.cc:2393] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5ed785fa1622433e8993e90b8c6c984e in current term 1: Already voted for candidate 153c2de7193c4844b765382a198e9db2 in this term.
I20260321 02:57:49.282021 10846 leader_election.cc:304] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5ed785fa1622433e8993e90b8c6c984e; no voters: 153c2de7193c4844b765382a198e9db2, 9f26c8327bfe4a58ae8b67940c9f49f8
I20260321 02:57:49.283591 10969 raft_consensus.cc:3060] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:57:49.289503 10733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "60d9453110c241f4aaa3740c3a43e6b6"
dest_uuid: "153c2de7193c4844b765382a198e9db2"
mode: GRACEFUL
new_leader_uuid: "5ed785fa1622433e8993e90b8c6c984e"
from {username='slave'} at 127.0.0.1:59914
I20260321 02:57:49.290308 10733 raft_consensus.cc:606] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Received request to transfer leadership to TS 5ed785fa1622433e8993e90b8c6c984e
I20260321 02:57:49.294412 11005 raft_consensus.cc:993] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e: : Instructing follower 9f26c8327bfe4a58ae8b67940c9f49f8 to start an election
I20260321 02:57:49.294569 10931 consensus_queue.cc:1048] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Connected to new peer: Peer: permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:49.294886 10970 raft_consensus.cc:1081] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 1 LEADER]: Signalling peer 9f26c8327bfe4a58ae8b67940c9f49f8 to start an election
I20260321 02:57:49.297402 10969 raft_consensus.cc:2749] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:57:49.298627 10807 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a56d5d5429b743bc8d2cb6fcb999f86e"
dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
from {username='slave'} at 127.0.0.1:37802
I20260321 02:57:49.299788 10807 raft_consensus.cc:493] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:49.300336 10807 raft_consensus.cc:3060] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:49.310446 10807 raft_consensus.cc:515] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:49.313465 10807 leader_election.cc:290] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 2 election: Requested vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:49.313354 10732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56d5d5429b743bc8d2cb6fcb999f86e" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "153c2de7193c4844b765382a198e9db2"
I20260321 02:57:49.314244 10732 raft_consensus.cc:3060] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:49.315078 10881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56d5d5429b743bc8d2cb6fcb999f86e" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
I20260321 02:57:49.315701 10881 raft_consensus.cc:3055] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:57:49.316061 10881 raft_consensus.cc:740] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Running, Role: LEADER
I20260321 02:57:49.316844 10881 consensus_queue.cc:260] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:49.318179 10881 raft_consensus.cc:3060] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:49.321643 10732 raft_consensus.cc:2468] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in term 2.
I20260321 02:57:49.324553 10771 leader_election.cc:304] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters:
I20260321 02:57:49.325415 10994 raft_consensus.cc:2804] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:57:49.325536 10881 raft_consensus.cc:2468] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in term 2.
I20260321 02:57:49.325922 10994 raft_consensus.cc:697] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 LEADER]: Becoming Leader. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Running, Role: LEADER
I20260321 02:57:49.326762 10994 consensus_queue.cc:237] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:49.333760 10629 catalog_manager.cc:5671] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 reported cstate change: term changed from 1 to 2, leader changed from 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3) to 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2). New cstate: current_term: 2 leader_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:49.337587 10964 consensus_queue.cc:1048] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:49.350174 10927 consensus_queue.cc:1048] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:49.356701 11005 consensus_queue.cc:1048] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:49.358914 10964 raft_consensus.cc:993] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2: : Instructing follower 9f26c8327bfe4a58ae8b67940c9f49f8 to start an election
I20260321 02:57:49.359439 10927 raft_consensus.cc:1081] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Signalling peer 9f26c8327bfe4a58ae8b67940c9f49f8 to start an election
I20260321 02:57:49.361109 10808 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b2633115df76454eb4c74d33d2b0b8fe"
dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
from {username='slave'} at 127.0.0.1:37790
I20260321 02:57:49.361718 10808 raft_consensus.cc:493] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:49.362382 10808 raft_consensus.cc:3060] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:49.368371 10969 raft_consensus.cc:993] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e: : Instructing follower 9f26c8327bfe4a58ae8b67940c9f49f8 to start an election
I20260321 02:57:49.368969 10931 raft_consensus.cc:1081] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 1 LEADER]: Signalling peer 9f26c8327bfe4a58ae8b67940c9f49f8 to start an election
I20260321 02:57:49.368639 10808 raft_consensus.cc:515] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:49.371642 10732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2633115df76454eb4c74d33d2b0b8fe" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "153c2de7193c4844b765382a198e9db2"
I20260321 02:57:49.372300 10732 raft_consensus.cc:3055] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:57:49.372625 10732 raft_consensus.cc:740] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Running, Role: LEADER
I20260321 02:57:49.373442 10808 leader_election.cc:290] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 2 election: Requested vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:49.373410 10732 consensus_queue.cc:260] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:49.374665 10732 raft_consensus.cc:3060] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:49.375661 10808 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e1d08251e74b419599f24a910bc9e538"
dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
from {username='slave'} at 127.0.0.1:37802
I20260321 02:57:49.376494 10881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2633115df76454eb4c74d33d2b0b8fe" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
I20260321 02:57:49.377007 11005 consensus_queue.cc:1048] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Connected to new peer: Peer: permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:49.377156 10881 raft_consensus.cc:3060] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:49.381850 10732 raft_consensus.cc:2468] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in term 2.
I20260321 02:57:49.379181 10808 raft_consensus.cc:493] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:49.383193 10771 leader_election.cc:304] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters:
I20260321 02:57:49.388234 10808 raft_consensus.cc:3060] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:49.388723 10994 raft_consensus.cc:2804] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:57:49.389281 10994 raft_consensus.cc:697] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 LEADER]: Becoming Leader. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Running, Role: LEADER
I20260321 02:57:49.389273 10881 raft_consensus.cc:2468] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in term 2.
I20260321 02:57:49.390131 10994 consensus_queue.cc:237] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:49.393339 10927 consensus_queue.cc:1048] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:49.399837 10808 raft_consensus.cc:515] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:49.402554 10808 leader_election.cc:290] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 2 election: Requested vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:49.403510 10733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d08251e74b419599f24a910bc9e538" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "153c2de7193c4844b765382a198e9db2"
I20260321 02:57:49.404189 10733 raft_consensus.cc:3060] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:49.405682 10881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d08251e74b419599f24a910bc9e538" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
I20260321 02:57:49.406423 10881 raft_consensus.cc:3055] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:57:49.406806 10881 raft_consensus.cc:740] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Running, Role: LEADER
I20260321 02:57:49.407737 10881 consensus_queue.cc:260] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:49.409202 10881 raft_consensus.cc:3060] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:49.410990 10954 consensus_queue.cc:1048] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:49.414502 10627 catalog_manager.cc:5671] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 reported cstate change: term changed from 1 to 2, leader changed from 153c2de7193c4844b765382a198e9db2 (127.9.213.1) to 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2). New cstate: current_term: 2 leader_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:49.417716 10881 raft_consensus.cc:2468] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in term 2.
I20260321 02:57:49.418045 10733 raft_consensus.cc:2468] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in term 2.
I20260321 02:57:49.418943 10771 leader_election.cc:304] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ed785fa1622433e8993e90b8c6c984e, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters:
I20260321 02:57:49.419757 10994 raft_consensus.cc:2804] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:57:49.420274 10994 raft_consensus.cc:697] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 LEADER]: Becoming Leader. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Running, Role: LEADER
I20260321 02:57:49.421116 10994 consensus_queue.cc:237] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:49.441799 10631 catalog_manager.cc:5671] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 reported cstate change: term changed from 1 to 2, leader changed from 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3) to 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2). New cstate: current_term: 2 leader_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:49.505172 10994 raft_consensus.cc:993] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8: : Instructing follower 153c2de7193c4844b765382a198e9db2 to start an election
I20260321 02:57:49.505663 10948 raft_consensus.cc:1081] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 LEADER]: Signalling peer 153c2de7193c4844b765382a198e9db2 to start an election
I20260321 02:57:49.507115 10733 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6e161a85964742b09837e0ae3dda87ca"
dest_uuid: "153c2de7193c4844b765382a198e9db2"
from {username='slave'} at 127.0.0.1:59898
I20260321 02:57:49.507622 10733 raft_consensus.cc:493] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:49.507926 10733 raft_consensus.cc:3060] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:49.512969 10733 raft_consensus.cc:515] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:49.514744 10733 leader_election.cc:290] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 2 election: Requested vote from peers 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:49.515714 10808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e161a85964742b09837e0ae3dda87ca" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
I20260321 02:57:49.516110 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e161a85964742b09837e0ae3dda87ca" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
I20260321 02:57:49.516672 10808 raft_consensus.cc:3055] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:57:49.516695 10882 raft_consensus.cc:3060] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:49.517190 10808 raft_consensus.cc:740] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Running, Role: LEADER
I20260321 02:57:49.517995 10808 consensus_queue.cc:260] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:49.519802 10808 raft_consensus.cc:3060] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:49.523655 10882 raft_consensus.cc:2468] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 2.
I20260321 02:57:49.525285 10696 leader_election.cc:304] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 5ed785fa1622433e8993e90b8c6c984e; no voters:
I20260321 02:57:49.525170 10964 consensus_queue.cc:1048] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:57:49.526068 10991 raft_consensus.cc:2804] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:57:49.526501 10991 raft_consensus.cc:697] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 2 LEADER]: Becoming Leader. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Running, Role: LEADER
I20260321 02:57:49.526538 10808 raft_consensus.cc:2468] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 2.
I20260321 02:57:49.527398 10991 consensus_queue.cc:237] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:49.534503 10954 raft_consensus.cc:993] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2: : Instructing follower 5ed785fa1622433e8993e90b8c6c984e to start an election
I20260321 02:57:49.534969 10954 raft_consensus.cc:1081] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Signalling peer 5ed785fa1622433e8993e90b8c6c984e to start an election
I20260321 02:57:49.539044 10627 catalog_manager.cc:5671] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 reported cstate change: term changed from 1 to 2, leader changed from 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2) to 153c2de7193c4844b765382a198e9db2 (127.9.213.1). New cstate: current_term: 2 leader_uuid: "153c2de7193c4844b765382a198e9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:49.540940 10881 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "60d9453110c241f4aaa3740c3a43e6b6"
dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
from {username='slave'} at 127.0.0.1:34902
I20260321 02:57:49.541646 10881 raft_consensus.cc:493] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:49.542112 10881 raft_consensus.cc:3060] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:49.542142 10964 consensus_queue.cc:1048] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:57:49.552495 10881 raft_consensus.cc:515] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:49.555545 10808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60d9453110c241f4aaa3740c3a43e6b6" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
I20260321 02:57:49.555301 10733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60d9453110c241f4aaa3740c3a43e6b6" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "153c2de7193c4844b765382a198e9db2"
I20260321 02:57:49.556238 10808 raft_consensus.cc:3060] T 60d9453110c241f4aaa3740c3a43e6b6 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:49.556439 10733 raft_consensus.cc:3055] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:57:49.556861 10733 raft_consensus.cc:740] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Running, Role: LEADER
I20260321 02:57:49.557639 10733 consensus_queue.cc:260] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:49.558511 10733 raft_consensus.cc:3060] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:49.560119 10881 leader_election.cc:290] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 2 election: Requested vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303)
I20260321 02:57:49.563871 10808 raft_consensus.cc:2468] T 60d9453110c241f4aaa3740c3a43e6b6 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed785fa1622433e8993e90b8c6c984e in term 2.
I20260321 02:57:49.564492 10733 raft_consensus.cc:2468] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed785fa1622433e8993e90b8c6c984e in term 2.
I20260321 02:57:49.565174 10846 leader_election.cc:304] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ed785fa1622433e8993e90b8c6c984e, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters:
I20260321 02:57:49.565888 11005 raft_consensus.cc:2804] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:57:49.566411 11005 raft_consensus.cc:697] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e [term 2 LEADER]: Becoming Leader. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Running, Role: LEADER
I20260321 02:57:49.567437 11005 consensus_queue.cc:237] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:49.573740 10631 catalog_manager.cc:5671] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e reported cstate change: term changed from 1 to 2, leader changed from 153c2de7193c4844b765382a198e9db2 (127.9.213.1) to 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3). New cstate: current_term: 2 leader_uuid: "5ed785fa1622433e8993e90b8c6c984e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: HEALTHY } } }
I20260321 02:57:49.680948 10994 raft_consensus.cc:993] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8: : Instructing follower 153c2de7193c4844b765382a198e9db2 to start an election
I20260321 02:57:49.681406 10948 raft_consensus.cc:1081] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 LEADER]: Signalling peer 153c2de7193c4844b765382a198e9db2 to start an election
I20260321 02:57:49.682770 10733 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "08f9bfdb4a464783b9b5b88025e96c4b"
dest_uuid: "153c2de7193c4844b765382a198e9db2"
from {username='slave'} at 127.0.0.1:59898
I20260321 02:57:49.683245 10733 raft_consensus.cc:493] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:49.683525 10733 raft_consensus.cc:3060] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:49.688838 10733 raft_consensus.cc:515] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:49.690496 10733 leader_election.cc:290] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 2 election: Requested vote from peers 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:57:49.691553 10808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08f9bfdb4a464783b9b5b88025e96c4b" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
I20260321 02:57:49.691884 10881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08f9bfdb4a464783b9b5b88025e96c4b" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
I20260321 02:57:49.692221 10808 raft_consensus.cc:3055] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:57:49.692456 10881 raft_consensus.cc:3060] T 08f9bfdb4a464783b9b5b88025e96c4b P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:49.692562 10808 raft_consensus.cc:740] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Running, Role: LEADER
I20260321 02:57:49.693430 10808 consensus_queue.cc:260] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:49.694649 10808 raft_consensus.cc:3060] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:49.699615 10881 raft_consensus.cc:2468] T 08f9bfdb4a464783b9b5b88025e96c4b P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 2.
I20260321 02:57:49.700836 10696 leader_election.cc:304] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 5ed785fa1622433e8993e90b8c6c984e; no voters:
I20260321 02:57:49.701633 10808 raft_consensus.cc:2468] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 2.
I20260321 02:57:49.701749 10964 raft_consensus.cc:2804] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:57:49.702526 10964 raft_consensus.cc:697] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [term 2 LEADER]: Becoming Leader. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Running, Role: LEADER
I20260321 02:57:49.703395 10964 consensus_queue.cc:237] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:49.710518 10631 catalog_manager.cc:5671] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 reported cstate change: term changed from 1 to 2, leader changed from 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2) to 153c2de7193c4844b765382a198e9db2 (127.9.213.1). New cstate: current_term: 2 leader_uuid: "153c2de7193c4844b765382a198e9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260321 02:57:49.720642 10881 raft_consensus.cc:1275] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Refusing update from remote peer 9f26c8327bfe4a58ae8b67940c9f49f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:49.721844 10948 consensus_queue.cc:1048] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:49.730132 10733 raft_consensus.cc:1275] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Refusing update from remote peer 9f26c8327bfe4a58ae8b67940c9f49f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:49.731907 10948 consensus_queue.cc:1048] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:57:49.815793 10733 raft_consensus.cc:1275] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Refusing update from remote peer 9f26c8327bfe4a58ae8b67940c9f49f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:49.817085 10941 consensus_queue.cc:1048] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:49.825289 10881 raft_consensus.cc:1275] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Refusing update from remote peer 9f26c8327bfe4a58ae8b67940c9f49f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:49.826892 10948 consensus_queue.cc:1048] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:49.838598 10881 raft_consensus.cc:1275] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Refusing update from remote peer 9f26c8327bfe4a58ae8b67940c9f49f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:49.839929 10994 consensus_queue.cc:1048] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:49.847811 10733 raft_consensus.cc:1275] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Refusing update from remote peer 9f26c8327bfe4a58ae8b67940c9f49f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:49.849433 10948 consensus_queue.cc:1048] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:49.937310 10807 raft_consensus.cc:1275] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Refusing update from remote peer 153c2de7193c4844b765382a198e9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:49.938711 10964 consensus_queue.cc:1048] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:49.948060 10881 raft_consensus.cc:1275] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Refusing update from remote peer 153c2de7193c4844b765382a198e9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:49.950276 10954 consensus_queue.cc:1048] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:50.031457 10733 raft_consensus.cc:1275] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Refusing update from remote peer 5ed785fa1622433e8993e90b8c6c984e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:50.033357 11005 consensus_queue.cc:1048] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Connected to new peer: Peer: permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:50.042675 10807 raft_consensus.cc:1275] T 60d9453110c241f4aaa3740c3a43e6b6 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Refusing update from remote peer 5ed785fa1622433e8993e90b8c6c984e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:50.044281 11005 consensus_queue.cc:1048] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:50.083670 10807 raft_consensus.cc:1275] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Refusing update from remote peer 153c2de7193c4844b765382a198e9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:50.085316 10954 consensus_queue.cc:1048] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:50.093765 10881 raft_consensus.cc:1275] T 08f9bfdb4a464783b9b5b88025e96c4b P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Refusing update from remote peer 153c2de7193c4844b765382a198e9db2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:50.095351 10954 consensus_queue.cc:1048] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:57:52.326128 10068 auto_leader_rebalancer-test.cc:343] The leader distribution is
I20260321 02:57:52.326861 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:52.343142 10732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6e161a85964742b09837e0ae3dda87ca"
dest_uuid: "153c2de7193c4844b765382a198e9db2"
mode: GRACEFUL
new_leader_uuid: "5ed785fa1622433e8993e90b8c6c984e"
from {username='slave'} at 127.0.0.1:59926
I20260321 02:57:52.343736 10732 raft_consensus.cc:606] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 2 LEADER]: Received request to transfer leadership to TS 5ed785fa1622433e8993e90b8c6c984e
I20260321 02:57:52.346563 10732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f35df046e02643dab408161cee9f00a4"
dest_uuid: "153c2de7193c4844b765382a198e9db2"
mode: GRACEFUL
new_leader_uuid: "5ed785fa1622433e8993e90b8c6c984e"
from {username='slave'} at 127.0.0.1:59926
I20260321 02:57:52.347036 10732 raft_consensus.cc:606] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Received request to transfer leadership to TS 5ed785fa1622433e8993e90b8c6c984e
I20260321 02:57:52.348203 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260321 02:57:52.348614 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:52.595052 11018 raft_consensus.cc:993] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2: : Instructing follower 5ed785fa1622433e8993e90b8c6c984e to start an election
I20260321 02:57:52.595626 11021 raft_consensus.cc:1081] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Signalling peer 5ed785fa1622433e8993e90b8c6c984e to start an election
I20260321 02:57:52.597127 10882 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f35df046e02643dab408161cee9f00a4"
dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
from {username='slave'} at 127.0.0.1:34902
I20260321 02:57:52.597656 10882 raft_consensus.cc:493] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:52.598022 10882 raft_consensus.cc:3060] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:52.602747 10882 raft_consensus.cc:515] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:52.604417 10882 leader_election.cc:290] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 2 election: Requested vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303)
I20260321 02:57:52.605263 10732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f35df046e02643dab408161cee9f00a4" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "153c2de7193c4844b765382a198e9db2"
I20260321 02:57:52.605551 10807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f35df046e02643dab408161cee9f00a4" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
I20260321 02:57:52.605999 10732 raft_consensus.cc:3055] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:57:52.606053 10807 raft_consensus.cc:3060] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:52.606500 10732 raft_consensus.cc:740] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Running, Role: LEADER
I20260321 02:57:52.607291 10732 consensus_queue.cc:260] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:52.608592 10732 raft_consensus.cc:3060] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:57:52.613718 10807 raft_consensus.cc:2468] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed785fa1622433e8993e90b8c6c984e in term 2.
I20260321 02:57:52.615025 10846 leader_election.cc:304] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ed785fa1622433e8993e90b8c6c984e, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters:
I20260321 02:57:52.615474 10732 raft_consensus.cc:2468] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed785fa1622433e8993e90b8c6c984e in term 2.
I20260321 02:57:52.615810 11005 raft_consensus.cc:2804] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:57:52.617033 11005 raft_consensus.cc:697] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [term 2 LEADER]: Becoming Leader. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Running, Role: LEADER
I20260321 02:57:52.617828 11005 consensus_queue.cc:237] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:52.624879 10627 catalog_manager.cc:5671] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e reported cstate change: term changed from 1 to 2, leader changed from 153c2de7193c4844b765382a198e9db2 (127.9.213.1) to 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3). New cstate: current_term: 2 leader_uuid: "5ed785fa1622433e8993e90b8c6c984e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: HEALTHY } } }
I20260321 02:57:52.691215 11018 raft_consensus.cc:993] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2: : Instructing follower 5ed785fa1622433e8993e90b8c6c984e to start an election
I20260321 02:57:52.691712 11021 raft_consensus.cc:1081] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 2 LEADER]: Signalling peer 5ed785fa1622433e8993e90b8c6c984e to start an election
I20260321 02:57:52.693310 10882 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6e161a85964742b09837e0ae3dda87ca"
dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
from {username='slave'} at 127.0.0.1:34902
I20260321 02:57:52.693926 10882 raft_consensus.cc:493] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:57:52.694373 10882 raft_consensus.cc:3060] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:52.699172 10882 raft_consensus.cc:515] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:52.700840 10882 leader_election.cc:290] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 3 election: Requested vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303)
I20260321 02:57:52.701824 10732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e161a85964742b09837e0ae3dda87ca" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "153c2de7193c4844b765382a198e9db2"
I20260321 02:57:52.701941 10807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e161a85964742b09837e0ae3dda87ca" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
I20260321 02:57:52.702356 10732 raft_consensus.cc:3055] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:57:52.702458 10807 raft_consensus.cc:3060] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:52.702658 10732 raft_consensus.cc:740] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Running, Role: LEADER
I20260321 02:57:52.703500 10732 consensus_queue.cc:260] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:52.704530 10732 raft_consensus.cc:3060] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:57:52.707510 10807 raft_consensus.cc:2468] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed785fa1622433e8993e90b8c6c984e in term 3.
I20260321 02:57:52.708539 10846 leader_election.cc:304] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ed785fa1622433e8993e90b8c6c984e, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters:
I20260321 02:57:52.709237 11005 raft_consensus.cc:2804] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:57:52.709420 10732 raft_consensus.cc:2468] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed785fa1622433e8993e90b8c6c984e in term 3.
I20260321 02:57:52.709787 11005 raft_consensus.cc:697] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 3 LEADER]: Becoming Leader. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Running, Role: LEADER
I20260321 02:57:52.710628 11005 consensus_queue.cc:237] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:57:52.717382 10627 catalog_manager.cc:5671] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e reported cstate change: term changed from 2 to 3, leader changed from 153c2de7193c4844b765382a198e9db2 (127.9.213.1) to 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3). New cstate: current_term: 3 leader_uuid: "5ed785fa1622433e8993e90b8c6c984e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: HEALTHY } } }
I20260321 02:57:53.053722 10732 raft_consensus.cc:1275] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Refusing update from remote peer 5ed785fa1622433e8993e90b8c6c984e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:53.055081 11005 consensus_queue.cc:1048] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Connected to new peer: Peer: permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:53.067014 10807 raft_consensus.cc:1275] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Refusing update from remote peer 5ed785fa1622433e8993e90b8c6c984e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:57:53.068542 11005 consensus_queue.cc:1048] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:57:53.170612 10733 raft_consensus.cc:1275] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 3 FOLLOWER]: Refusing update from remote peer 5ed785fa1622433e8993e90b8c6c984e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:57:53.171912 11005 consensus_queue.cc:1048] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Connected to new peer: Peer: permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:57:53.182721 10807 raft_consensus.cc:1275] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 3 FOLLOWER]: Refusing update from remote peer 5ed785fa1622433e8993e90b8c6c984e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:57:53.184378 11036 consensus_queue.cc:1048] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:57:53.349444 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:53.354797 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:53.355168 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:54.356510 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:54.363283 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:54.363804 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:55.364900 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:55.372282 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:55.372733 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:56.373919 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:56.380107 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:56.380522 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:57.381567 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:57.387907 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:57.388372 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:58.389377 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:58.397059 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:58.397485 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:57:59.398404 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:57:59.404899 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:57:59.405301 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:58:00.406564 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:58:00.412909 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:58:00.413512 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:58:01.414547 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:58:01.421499 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:58:01.421898 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:58:02.461319 10068 auto_leader_rebalancer-test.cc:362] The leader distribution is
I20260321 02:58:02.467578 10882 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6e161a85964742b09837e0ae3dda87ca"
dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
mode: GRACEFUL
new_leader_uuid: "153c2de7193c4844b765382a198e9db2"
from {username='slave'} at 127.0.0.1:34908
I20260321 02:58:02.468248 10882 raft_consensus.cc:606] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 3 LEADER]: Received request to transfer leadership to TS 153c2de7193c4844b765382a198e9db2
I20260321 02:58:02.484930 10882 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f35df046e02643dab408161cee9f00a4"
dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
mode: GRACEFUL
new_leader_uuid: "153c2de7193c4844b765382a198e9db2"
from {username='slave'} at 127.0.0.1:34908
I20260321 02:58:02.485402 10882 raft_consensus.cc:606] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [term 2 LEADER]: Received request to transfer leadership to TS 153c2de7193c4844b765382a198e9db2
I20260321 02:58:02.491899 10808 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a56d5d5429b743bc8d2cb6fcb999f86e"
dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
mode: GRACEFUL
new_leader_uuid: "153c2de7193c4844b765382a198e9db2"
from {username='slave'} at 127.0.0.1:37816
I20260321 02:58:02.492403 10808 raft_consensus.cc:606] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 LEADER]: Received request to transfer leadership to TS 153c2de7193c4844b765382a198e9db2
I20260321 02:58:02.499791 10808 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e1d08251e74b419599f24a910bc9e538"
dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
mode: GRACEFUL
new_leader_uuid: "153c2de7193c4844b765382a198e9db2"
from {username='slave'} at 127.0.0.1:37816
I20260321 02:58:02.500474 10808 raft_consensus.cc:606] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 LEADER]: Received request to transfer leadership to TS 153c2de7193c4844b765382a198e9db2
I20260321 02:58:02.506237 10808 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b2633115df76454eb4c74d33d2b0b8fe"
dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
mode: GRACEFUL
new_leader_uuid: "153c2de7193c4844b765382a198e9db2"
from {username='slave'} at 127.0.0.1:37816
I20260321 02:58:02.506767 10808 raft_consensus.cc:606] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 LEADER]: Received request to transfer leadership to TS 153c2de7193c4844b765382a198e9db2
I20260321 02:58:02.522840 11022 raft_consensus.cc:993] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e: : Instructing follower 153c2de7193c4844b765382a198e9db2 to start an election
I20260321 02:58:02.523234 11022 raft_consensus.cc:1081] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 3 LEADER]: Signalling peer 153c2de7193c4844b765382a198e9db2 to start an election
I20260321 02:58:02.525949 10733 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6e161a85964742b09837e0ae3dda87ca"
dest_uuid: "153c2de7193c4844b765382a198e9db2"
from {username='slave'} at 127.0.0.1:59900
I20260321 02:58:02.526499 10733 raft_consensus.cc:493] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:58:02.526854 10733 raft_consensus.cc:3060] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:02.535750 10733 raft_consensus.cc:515] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:02.537883 10733 leader_election.cc:290] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 4 election: Requested vote from peers 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:58:02.539402 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e161a85964742b09837e0ae3dda87ca" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
I20260321 02:58:02.539582 10808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e161a85964742b09837e0ae3dda87ca" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
I20260321 02:58:02.540103 10882 raft_consensus.cc:3055] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:58:02.540154 10808 raft_consensus.cc:3060] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:02.540457 10882 raft_consensus.cc:740] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Running, Role: LEADER
I20260321 02:58:02.541857 10882 consensus_queue.cc:260] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:02.543082 10882 raft_consensus.cc:3060] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:02.549170 10808 raft_consensus.cc:2468] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 4.
I20260321 02:58:02.550928 10697 leader_election.cc:304] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters:
I20260321 02:58:02.551860 11021 raft_consensus.cc:2804] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:58:02.552865 11021 raft_consensus.cc:697] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 4 LEADER]: Becoming Leader. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Running, Role: LEADER
I20260321 02:58:02.552990 10882 raft_consensus.cc:2468] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 4.
I20260321 02:58:02.553731 11021 consensus_queue.cc:237] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:02.561378 10631 catalog_manager.cc:5671] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 reported cstate change: term changed from 3 to 4, leader changed from 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3) to 153c2de7193c4844b765382a198e9db2 (127.9.213.1). New cstate: current_term: 4 leader_uuid: "153c2de7193c4844b765382a198e9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:02.624771 11052 raft_consensus.cc:993] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8: : Instructing follower 153c2de7193c4844b765382a198e9db2 to start an election
I20260321 02:58:02.625294 11044 raft_consensus.cc:1081] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 LEADER]: Signalling peer 153c2de7193c4844b765382a198e9db2 to start an election
I20260321 02:58:02.627518 10733 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a56d5d5429b743bc8d2cb6fcb999f86e"
dest_uuid: "153c2de7193c4844b765382a198e9db2"
from {username='slave'} at 127.0.0.1:59898
I20260321 02:58:02.628137 10733 raft_consensus.cc:493] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:58:02.628427 10733 raft_consensus.cc:3060] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:02.633090 10733 raft_consensus.cc:515] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:02.634589 10733 leader_election.cc:290] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 3 election: Requested vote from peers 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:58:02.635488 10808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56d5d5429b743bc8d2cb6fcb999f86e" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
I20260321 02:58:02.635591 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a56d5d5429b743bc8d2cb6fcb999f86e" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
I20260321 02:58:02.636058 10808 raft_consensus.cc:3055] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:58:02.636153 10882 raft_consensus.cc:3060] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:02.636344 10808 raft_consensus.cc:740] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Running, Role: LEADER
I20260321 02:58:02.637111 10808 consensus_queue.cc:260] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:02.638006 10808 raft_consensus.cc:3060] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:02.641073 10882 raft_consensus.cc:2468] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 3.
I20260321 02:58:02.642421 10696 leader_election.cc:304] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 5ed785fa1622433e8993e90b8c6c984e; no voters:
I20260321 02:58:02.642763 10808 raft_consensus.cc:2468] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 3.
I20260321 02:58:02.643671 11021 raft_consensus.cc:2804] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:02.645376 11021 raft_consensus.cc:697] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [term 3 LEADER]: Becoming Leader. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Running, Role: LEADER
I20260321 02:58:02.646242 11021 consensus_queue.cc:237] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:02.655025 10631 catalog_manager.cc:5671] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 reported cstate change: term changed from 2 to 3, leader changed from 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2) to 153c2de7193c4844b765382a198e9db2 (127.9.213.1). New cstate: current_term: 3 leader_uuid: "153c2de7193c4844b765382a198e9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:02.669776 11045 raft_consensus.cc:993] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e: : Instructing follower 153c2de7193c4844b765382a198e9db2 to start an election
I20260321 02:58:02.670343 11045 raft_consensus.cc:1081] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [term 2 LEADER]: Signalling peer 153c2de7193c4844b765382a198e9db2 to start an election
I20260321 02:58:02.672081 10733 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f35df046e02643dab408161cee9f00a4"
dest_uuid: "153c2de7193c4844b765382a198e9db2"
from {username='slave'} at 127.0.0.1:59900
I20260321 02:58:02.672616 10733 raft_consensus.cc:493] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:58:02.672958 10733 raft_consensus.cc:3060] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:02.677418 10733 raft_consensus.cc:515] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:02.678773 10733 leader_election.cc:290] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 3 election: Requested vote from peers 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:58:02.679836 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f35df046e02643dab408161cee9f00a4" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
I20260321 02:58:02.680805 10882 raft_consensus.cc:3055] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:58:02.680840 10808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f35df046e02643dab408161cee9f00a4" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
I20260321 02:58:02.681437 10882 raft_consensus.cc:740] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Running, Role: LEADER
I20260321 02:58:02.681692 10808 raft_consensus.cc:3060] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:02.682488 10882 consensus_queue.cc:260] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:02.684257 10882 raft_consensus.cc:3060] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:02.690096 10808 raft_consensus.cc:2468] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 3.
I20260321 02:58:02.691221 10882 raft_consensus.cc:2468] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 3.
I20260321 02:58:02.691733 10697 leader_election.cc:304] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters:
I20260321 02:58:02.692873 11021 raft_consensus.cc:2804] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:02.693311 11021 raft_consensus.cc:697] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 3 LEADER]: Becoming Leader. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Running, Role: LEADER
I20260321 02:58:02.694293 11021 consensus_queue.cc:237] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:02.703910 11052 raft_consensus.cc:993] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8: : Instructing follower 153c2de7193c4844b765382a198e9db2 to start an election
I20260321 02:58:02.704635 11044 raft_consensus.cc:1081] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 LEADER]: Signalling peer 153c2de7193c4844b765382a198e9db2 to start an election
I20260321 02:58:02.703950 10631 catalog_manager.cc:5671] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 reported cstate change: term changed from 2 to 3, leader changed from 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3) to 153c2de7193c4844b765382a198e9db2 (127.9.213.1). New cstate: current_term: 3 leader_uuid: "153c2de7193c4844b765382a198e9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:02.707417 10733 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b2633115df76454eb4c74d33d2b0b8fe"
dest_uuid: "153c2de7193c4844b765382a198e9db2"
from {username='slave'} at 127.0.0.1:59898
I20260321 02:58:02.708446 10733 raft_consensus.cc:493] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:58:02.708986 10733 raft_consensus.cc:3060] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:02.716046 10733 raft_consensus.cc:515] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:02.718277 10733 leader_election.cc:290] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 3 election: Requested vote from peers 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:58:02.719352 10808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2633115df76454eb4c74d33d2b0b8fe" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
I20260321 02:58:02.719910 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2633115df76454eb4c74d33d2b0b8fe" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
I20260321 02:58:02.720101 10808 raft_consensus.cc:3055] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:58:02.720440 10882 raft_consensus.cc:3060] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:02.720408 10808 raft_consensus.cc:740] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Running, Role: LEADER
I20260321 02:58:02.721509 10808 consensus_queue.cc:260] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:02.722883 10808 raft_consensus.cc:3060] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:02.728408 10882 raft_consensus.cc:2468] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 3.
I20260321 02:58:02.730054 10696 leader_election.cc:304] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 5ed785fa1622433e8993e90b8c6c984e; no voters:
I20260321 02:58:02.730604 10808 raft_consensus.cc:2468] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 3.
I20260321 02:58:02.730851 11021 raft_consensus.cc:2804] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:02.731264 11021 raft_consensus.cc:697] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 3 LEADER]: Becoming Leader. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Running, Role: LEADER
I20260321 02:58:02.732088 11021 consensus_queue.cc:237] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:02.741123 10631 catalog_manager.cc:5671] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 reported cstate change: term changed from 2 to 3, leader changed from 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2) to 153c2de7193c4844b765382a198e9db2 (127.9.213.1). New cstate: current_term: 3 leader_uuid: "153c2de7193c4844b765382a198e9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:02.743305 11052 raft_consensus.cc:993] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8: : Instructing follower 153c2de7193c4844b765382a198e9db2 to start an election
I20260321 02:58:02.744088 11044 raft_consensus.cc:1081] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 LEADER]: Signalling peer 153c2de7193c4844b765382a198e9db2 to start an election
I20260321 02:58:02.745724 10733 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e1d08251e74b419599f24a910bc9e538"
dest_uuid: "153c2de7193c4844b765382a198e9db2"
from {username='slave'} at 127.0.0.1:59898
I20260321 02:58:02.746206 10733 raft_consensus.cc:493] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:58:02.746636 10733 raft_consensus.cc:3060] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:02.752861 10733 raft_consensus.cc:515] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:02.755482 10733 leader_election.cc:290] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 3 election: Requested vote from peers 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:58:02.756858 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d08251e74b419599f24a910bc9e538" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
I20260321 02:58:02.757342 10882 raft_consensus.cc:3060] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:02.757347 10808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d08251e74b419599f24a910bc9e538" candidate_uuid: "153c2de7193c4844b765382a198e9db2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
I20260321 02:58:02.758199 10808 raft_consensus.cc:3055] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:58:02.759145 10808 raft_consensus.cc:740] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Running, Role: LEADER
I20260321 02:58:02.761184 10808 consensus_queue.cc:260] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:02.762157 10808 raft_consensus.cc:3060] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:02.764699 10882 raft_consensus.cc:2468] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 3.
I20260321 02:58:02.766062 10696 leader_election.cc:304] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153c2de7193c4844b765382a198e9db2, 5ed785fa1622433e8993e90b8c6c984e; no voters:
I20260321 02:58:02.767163 11021 raft_consensus.cc:2804] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:02.767704 10808 raft_consensus.cc:2468] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153c2de7193c4844b765382a198e9db2 in term 3.
I20260321 02:58:02.768092 11021 raft_consensus.cc:697] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 3 LEADER]: Becoming Leader. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Running, Role: LEADER
I20260321 02:58:02.769819 11021 consensus_queue.cc:237] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:02.776722 10631 catalog_manager.cc:5671] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 reported cstate change: term changed from 2 to 3, leader changed from 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2) to 153c2de7193c4844b765382a198e9db2 (127.9.213.1). New cstate: current_term: 3 leader_uuid: "153c2de7193c4844b765382a198e9db2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:03.043686 10808 raft_consensus.cc:1275] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 3 FOLLOWER]: Refusing update from remote peer 153c2de7193c4844b765382a198e9db2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:58:03.044965 11021 consensus_queue.cc:1048] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:03.060141 10882 raft_consensus.cc:1275] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 3 FOLLOWER]: Refusing update from remote peer 153c2de7193c4844b765382a198e9db2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:58:03.062405 11053 consensus_queue.cc:1048] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:03.075608 10808 raft_consensus.cc:1275] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 3 FOLLOWER]: Refusing update from remote peer 153c2de7193c4844b765382a198e9db2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:58:03.076875 11053 consensus_queue.cc:1048] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:03.082214 10882 raft_consensus.cc:1275] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 4 FOLLOWER]: Refusing update from remote peer 153c2de7193c4844b765382a198e9db2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 02:58:03.084057 11053 consensus_queue.cc:1048] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:03.088505 10881 raft_consensus.cc:1275] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [term 3 FOLLOWER]: Refusing update from remote peer 153c2de7193c4844b765382a198e9db2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:58:03.090396 11046 consensus_queue.cc:1048] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:03.107641 10808 raft_consensus.cc:1275] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 4 FOLLOWER]: Refusing update from remote peer 153c2de7193c4844b765382a198e9db2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 02:58:03.109346 11021 consensus_queue.cc:1048] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:03.257315 10882 raft_consensus.cc:1275] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 3 FOLLOWER]: Refusing update from remote peer 153c2de7193c4844b765382a198e9db2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:58:03.258720 11021 consensus_queue.cc:1048] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:03.268949 10808 raft_consensus.cc:1275] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 3 FOLLOWER]: Refusing update from remote peer 153c2de7193c4844b765382a198e9db2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:58:03.270623 11064 consensus_queue.cc:1048] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:03.285797 10808 raft_consensus.cc:1275] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 3 FOLLOWER]: Refusing update from remote peer 153c2de7193c4844b765382a198e9db2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:58:03.287185 11064 consensus_queue.cc:1048] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:03.297410 10882 raft_consensus.cc:1275] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 3 FOLLOWER]: Refusing update from remote peer 153c2de7193c4844b765382a198e9db2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:58:03.300158 11021 consensus_queue.cc:1048] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:05.552877 10068 auto_leader_rebalancer-test.cc:377] The leader distribution is
I20260321 02:58:05.553568 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:58:05.561098 10733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "082582ef8a7a483eaf98b4ced6214a57"
dest_uuid: "153c2de7193c4844b765382a198e9db2"
mode: GRACEFUL
new_leader_uuid: "5ed785fa1622433e8993e90b8c6c984e"
from {username='slave'} at 127.0.0.1:59926
I20260321 02:58:05.561807 10733 raft_consensus.cc:606] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Received request to transfer leadership to TS 5ed785fa1622433e8993e90b8c6c984e
I20260321 02:58:05.564270 10733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3a1c86bd318d452e9446f7ab8783c20f"
dest_uuid: "153c2de7193c4844b765382a198e9db2"
mode: GRACEFUL
new_leader_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
from {username='slave'} at 127.0.0.1:59926
I20260321 02:58:05.564728 10733 raft_consensus.cc:606] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Received request to transfer leadership to TS 9f26c8327bfe4a58ae8b67940c9f49f8
I20260321 02:58:05.566814 10733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b2633115df76454eb4c74d33d2b0b8fe"
dest_uuid: "153c2de7193c4844b765382a198e9db2"
mode: GRACEFUL
new_leader_uuid: "5ed785fa1622433e8993e90b8c6c984e"
from {username='slave'} at 127.0.0.1:59926
I20260321 02:58:05.567185 10733 raft_consensus.cc:606] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 3 LEADER]: Received request to transfer leadership to TS 5ed785fa1622433e8993e90b8c6c984e
I20260321 02:58:05.569358 10733 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e1d08251e74b419599f24a910bc9e538"
dest_uuid: "153c2de7193c4844b765382a198e9db2"
mode: GRACEFUL
new_leader_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
from {username='slave'} at 127.0.0.1:59926
I20260321 02:58:05.569797 10733 raft_consensus.cc:606] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 3 LEADER]: Received request to transfer leadership to TS 9f26c8327bfe4a58ae8b67940c9f49f8
I20260321 02:58:05.570802 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260321 02:58:05.571254 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:58:05.721506 11046 raft_consensus.cc:993] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2: : Instructing follower 5ed785fa1622433e8993e90b8c6c984e to start an election
I20260321 02:58:05.721992 11046 raft_consensus.cc:1081] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 3 LEADER]: Signalling peer 5ed785fa1622433e8993e90b8c6c984e to start an election
I20260321 02:58:05.723546 10882 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b2633115df76454eb4c74d33d2b0b8fe"
dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
from {username='slave'} at 127.0.0.1:34902
I20260321 02:58:05.724198 10882 raft_consensus.cc:493] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:58:05.724530 10882 raft_consensus.cc:3060] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:05.728938 10882 raft_consensus.cc:515] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:05.731173 10882 leader_election.cc:290] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 4 election: Requested vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303)
I20260321 02:58:05.732220 10733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2633115df76454eb4c74d33d2b0b8fe" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "153c2de7193c4844b765382a198e9db2"
I20260321 02:58:05.732326 10807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2633115df76454eb4c74d33d2b0b8fe" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
I20260321 02:58:05.732764 10733 raft_consensus.cc:3055] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:58:05.732882 10807 raft_consensus.cc:3060] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:05.733089 10733 raft_consensus.cc:740] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Running, Role: LEADER
I20260321 02:58:05.733834 10733 consensus_queue.cc:260] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:05.734712 10733 raft_consensus.cc:3060] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:05.738837 10807 raft_consensus.cc:2468] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed785fa1622433e8993e90b8c6c984e in term 4.
I20260321 02:58:05.740314 10846 leader_election.cc:304] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ed785fa1622433e8993e90b8c6c984e, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters:
I20260321 02:58:05.741216 11045 raft_consensus.cc:2804] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:58:05.741322 10733 raft_consensus.cc:2468] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed785fa1622433e8993e90b8c6c984e in term 4.
I20260321 02:58:05.742365 11045 raft_consensus.cc:697] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 4 LEADER]: Becoming Leader. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Running, Role: LEADER
I20260321 02:58:05.743352 11045 consensus_queue.cc:237] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:05.750931 10627 catalog_manager.cc:5671] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e reported cstate change: term changed from 3 to 4, leader changed from 153c2de7193c4844b765382a198e9db2 (127.9.213.1) to 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3). New cstate: current_term: 4 leader_uuid: "5ed785fa1622433e8993e90b8c6c984e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:05.811874 11073 raft_consensus.cc:993] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2: : Instructing follower 9f26c8327bfe4a58ae8b67940c9f49f8 to start an election
I20260321 02:58:05.812606 11046 raft_consensus.cc:1081] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Signalling peer 9f26c8327bfe4a58ae8b67940c9f49f8 to start an election
I20260321 02:58:05.814085 10807 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3a1c86bd318d452e9446f7ab8783c20f"
dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
from {username='slave'} at 127.0.0.1:37790
I20260321 02:58:05.814594 10807 raft_consensus.cc:493] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:58:05.814883 10807 raft_consensus.cc:3060] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:05.819497 10807 raft_consensus.cc:515] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:05.821506 10807 leader_election.cc:290] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 2 election: Requested vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:58:05.822567 10733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a1c86bd318d452e9446f7ab8783c20f" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "153c2de7193c4844b765382a198e9db2"
I20260321 02:58:05.822667 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a1c86bd318d452e9446f7ab8783c20f" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
I20260321 02:58:05.823146 10733 raft_consensus.cc:3055] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:05.823228 10882 raft_consensus.cc:3060] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:05.823524 10733 raft_consensus.cc:740] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Running, Role: LEADER
I20260321 02:58:05.824606 10733 consensus_queue.cc:260] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:05.825520 10733 raft_consensus.cc:3060] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:05.828651 10882 raft_consensus.cc:2468] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in term 2.
I20260321 02:58:05.829969 10733 raft_consensus.cc:2468] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in term 2.
I20260321 02:58:05.829969 10771 leader_election.cc:304] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ed785fa1622433e8993e90b8c6c984e, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters:
I20260321 02:58:05.830943 11044 raft_consensus.cc:2804] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:58:05.832114 11044 raft_consensus.cc:697] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 LEADER]: Becoming Leader. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Running, Role: LEADER
I20260321 02:58:05.833087 11044 consensus_queue.cc:237] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:05.841096 10627 catalog_manager.cc:5671] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 reported cstate change: term changed from 1 to 2, leader changed from 153c2de7193c4844b765382a198e9db2 (127.9.213.1) to 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2). New cstate: current_term: 2 leader_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:05.883524 11046 raft_consensus.cc:993] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2: : Instructing follower 5ed785fa1622433e8993e90b8c6c984e to start an election
I20260321 02:58:05.884008 11073 raft_consensus.cc:1081] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Signalling peer 5ed785fa1622433e8993e90b8c6c984e to start an election
I20260321 02:58:05.885596 10882 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "082582ef8a7a483eaf98b4ced6214a57"
dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
from {username='slave'} at 127.0.0.1:34902
I20260321 02:58:05.886237 10882 raft_consensus.cc:493] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:58:05.886776 10882 raft_consensus.cc:3060] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:05.891093 10882 raft_consensus.cc:515] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:05.893323 10882 leader_election.cc:290] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 2 election: Requested vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2:44303)
I20260321 02:58:05.894516 10733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "082582ef8a7a483eaf98b4ced6214a57" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "153c2de7193c4844b765382a198e9db2"
I20260321 02:58:05.894896 10807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "082582ef8a7a483eaf98b4ced6214a57" candidate_uuid: "5ed785fa1622433e8993e90b8c6c984e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
I20260321 02:58:05.895262 10733 raft_consensus.cc:3055] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:05.895357 10807 raft_consensus.cc:3060] T 082582ef8a7a483eaf98b4ced6214a57 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:05.895744 10733 raft_consensus.cc:740] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Running, Role: LEADER
I20260321 02:58:05.896819 10733 consensus_queue.cc:260] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:05.898032 10733 raft_consensus.cc:3060] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:05.902256 10807 raft_consensus.cc:2468] T 082582ef8a7a483eaf98b4ced6214a57 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed785fa1622433e8993e90b8c6c984e in term 2.
I20260321 02:58:05.903340 10846 leader_election.cc:304] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ed785fa1622433e8993e90b8c6c984e, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters:
I20260321 02:58:05.904213 11045 raft_consensus.cc:2804] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:58:05.904381 10733 raft_consensus.cc:2468] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ed785fa1622433e8993e90b8c6c984e in term 2.
I20260321 02:58:05.904706 11045 raft_consensus.cc:697] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [term 2 LEADER]: Becoming Leader. State: Replica: 5ed785fa1622433e8993e90b8c6c984e, State: Running, Role: LEADER
I20260321 02:58:05.905791 11045 consensus_queue.cc:237] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:05.939096 10627 catalog_manager.cc:5671] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e reported cstate change: term changed from 1 to 2, leader changed from 153c2de7193c4844b765382a198e9db2 (127.9.213.1) to 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3). New cstate: current_term: 2 leader_uuid: "5ed785fa1622433e8993e90b8c6c984e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:06.019280 11046 raft_consensus.cc:993] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2: : Instructing follower 9f26c8327bfe4a58ae8b67940c9f49f8 to start an election
I20260321 02:58:06.020041 11073 raft_consensus.cc:1081] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 3 LEADER]: Signalling peer 9f26c8327bfe4a58ae8b67940c9f49f8 to start an election
I20260321 02:58:06.021581 10808 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e1d08251e74b419599f24a910bc9e538"
dest_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8"
from {username='slave'} at 127.0.0.1:37790
I20260321 02:58:06.022135 10808 raft_consensus.cc:493] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:58:06.022470 10808 raft_consensus.cc:3060] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:06.028777 10808 raft_consensus.cc:515] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:06.030802 10808 leader_election.cc:290] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 4 election: Requested vote from peers 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123), 5ed785fa1622433e8993e90b8c6c984e (127.9.213.3:34785)
I20260321 02:58:06.032179 10882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d08251e74b419599f24a910bc9e538" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "5ed785fa1622433e8993e90b8c6c984e"
I20260321 02:58:06.032317 10733 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1d08251e74b419599f24a910bc9e538" candidate_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "153c2de7193c4844b765382a198e9db2"
I20260321 02:58:06.032897 10882 raft_consensus.cc:3060] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:06.033099 10733 raft_consensus.cc:3055] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:58:06.033634 10733 raft_consensus.cc:740] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 153c2de7193c4844b765382a198e9db2, State: Running, Role: LEADER
I20260321 02:58:06.034409 10733 consensus_queue.cc:260] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:06.035656 10733 raft_consensus.cc:3060] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:06.038337 10882 raft_consensus.cc:2468] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in term 4.
I20260321 02:58:06.039463 10771 leader_election.cc:304] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ed785fa1622433e8993e90b8c6c984e, 9f26c8327bfe4a58ae8b67940c9f49f8; no voters:
I20260321 02:58:06.040329 11044 raft_consensus.cc:2804] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:58:06.040812 11044 raft_consensus.cc:697] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 4 LEADER]: Becoming Leader. State: Replica: 9f26c8327bfe4a58ae8b67940c9f49f8, State: Running, Role: LEADER
I20260321 02:58:06.042030 10733 raft_consensus.cc:2468] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f26c8327bfe4a58ae8b67940c9f49f8 in term 4.
I20260321 02:58:06.041886 11044 consensus_queue.cc:237] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } }
I20260321 02:58:06.050388 10627 catalog_manager.cc:5671] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 reported cstate change: term changed from 3 to 4, leader changed from 153c2de7193c4844b765382a198e9db2 (127.9.213.1) to 9f26c8327bfe4a58ae8b67940c9f49f8 (127.9.213.2). New cstate: current_term: 4 leader_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:06.225224 10733 raft_consensus.cc:1275] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Refusing update from remote peer 9f26c8327bfe4a58ae8b67940c9f49f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:58:06.226780 11044 consensus_queue.cc:1048] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:06.239562 10882 raft_consensus.cc:1275] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Refusing update from remote peer 9f26c8327bfe4a58ae8b67940c9f49f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:58:06.240926 11080 consensus_queue.cc:1048] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:06.274188 10808 raft_consensus.cc:1275] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 4 FOLLOWER]: Refusing update from remote peer 5ed785fa1622433e8993e90b8c6c984e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 02:58:06.275743 11045 consensus_queue.cc:1048] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:06.286886 10733 raft_consensus.cc:1275] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 4 FOLLOWER]: Refusing update from remote peer 5ed785fa1622433e8993e90b8c6c984e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 02:58:06.288318 11045 consensus_queue.cc:1048] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Connected to new peer: Peer: permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:06.443300 10732 raft_consensus.cc:1275] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Refusing update from remote peer 5ed785fa1622433e8993e90b8c6c984e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:58:06.444800 11045 consensus_queue.cc:1048] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Connected to new peer: Peer: permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:06.456279 10808 raft_consensus.cc:1275] T 082582ef8a7a483eaf98b4ced6214a57 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Refusing update from remote peer 5ed785fa1622433e8993e90b8c6c984e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:58:06.458001 11045 consensus_queue.cc:1048] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f26c8327bfe4a58ae8b67940c9f49f8" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 44303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:06.466485 10882 raft_consensus.cc:1275] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 4 FOLLOWER]: Refusing update from remote peer 9f26c8327bfe4a58ae8b67940c9f49f8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 02:58:06.468433 11044 consensus_queue.cc:1048] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ed785fa1622433e8993e90b8c6c984e" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:06.481369 10732 raft_consensus.cc:1275] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 4 FOLLOWER]: Refusing update from remote peer 9f26c8327bfe4a58ae8b67940c9f49f8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 02:58:06.483832 11044 consensus_queue.cc:1048] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153c2de7193c4844b765382a198e9db2" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 42123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:06.572573 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:58:06.578944 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:58:06.579454 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:58:07.580626 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:58:07.587764 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:58:07.588378 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:58:08.589519 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:58:08.595095 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:58:08.595624 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:58:09.596652 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:58:09.602726 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:58:09.603202 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:58:10.604437 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:58:10.613488 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:58:10.613947 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:58:11.615509 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:58:11.621201 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:58:11.621840 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:58:12.622869 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:58:12.628257 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:58:12.628732 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:58:13.629737 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:58:13.636327 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:58:13.636955 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:58:14.637919 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:58:14.643059 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260321 02:58:14.643510 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:58:15.676066 10068 auto_leader_rebalancer-test.cc:388] The leader distribution is
I20260321 02:58:15.676584 10068 tablet_server.cc:179] TabletServer@127.9.213.1:0 shutting down...
W20260321 02:58:15.693819 10771 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 3 similar messages]
W20260321 02:58:15.697958 10771 consensus_peers.cc:597] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 -> Peer 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123): Couldn't send request to peer 153c2de7193c4844b765382a198e9db2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:15.708391 10068 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 02:58:15.709231 10068 tablet_replica.cc:333] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 153c2de7193c4844b765382a198e9db2: stopping tablet replica
I20260321 02:58:15.709825 10068 raft_consensus.cc:2243] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:15.710227 10068 raft_consensus.cc:2272] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 153c2de7193c4844b765382a198e9db2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.713465 10068 tablet_replica.cc:333] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2: stopping tablet replica
I20260321 02:58:15.713999 10068 raft_consensus.cc:2243] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:15.714445 10068 raft_consensus.cc:2272] T 082582ef8a7a483eaf98b4ced6214a57 P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.716197 10068 tablet_replica.cc:333] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2: stopping tablet replica
I20260321 02:58:15.716775 10068 raft_consensus.cc:2243] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:15.717231 10068 raft_consensus.cc:2272] T b2633115df76454eb4c74d33d2b0b8fe P 153c2de7193c4844b765382a198e9db2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.719463 10068 tablet_replica.cc:333] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2: stopping tablet replica
I20260321 02:58:15.720224 10068 raft_consensus.cc:2243] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:15.720707 10068 raft_consensus.cc:2272] T 3a1c86bd318d452e9446f7ab8783c20f P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.723006 10068 tablet_replica.cc:333] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2: stopping tablet replica
I20260321 02:58:15.723850 10068 raft_consensus.cc:2243] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [term 2 LEADER]: Raft consensus shutting down.
I20260321 02:58:15.724972 10068 raft_consensus.cc:2272] T 08f9bfdb4a464783b9b5b88025e96c4b P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.727501 10068 tablet_replica.cc:333] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2: stopping tablet replica
I20260321 02:58:15.728281 10068 raft_consensus.cc:2243] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [term 3 LEADER]: Raft consensus shutting down.
I20260321 02:58:15.729671 10068 raft_consensus.cc:2272] T a56d5d5429b743bc8d2cb6fcb999f86e P 153c2de7193c4844b765382a198e9db2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.732501 10068 tablet_replica.cc:333] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2: stopping tablet replica
I20260321 02:58:15.732959 10068 raft_consensus.cc:2243] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:15.733381 10068 raft_consensus.cc:2272] T 60d9453110c241f4aaa3740c3a43e6b6 P 153c2de7193c4844b765382a198e9db2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.735247 10068 tablet_replica.cc:333] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2: stopping tablet replica
I20260321 02:58:15.735826 10068 raft_consensus.cc:2243] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 4 LEADER]: Raft consensus shutting down.
I20260321 02:58:15.736804 10068 raft_consensus.cc:2272] T 6e161a85964742b09837e0ae3dda87ca P 153c2de7193c4844b765382a198e9db2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.738840 10068 tablet_replica.cc:333] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2: stopping tablet replica
I20260321 02:58:15.739329 10068 raft_consensus.cc:2243] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:15.739738 10068 raft_consensus.cc:2272] T e1d08251e74b419599f24a910bc9e538 P 153c2de7193c4844b765382a198e9db2 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.741380 10068 tablet_replica.cc:333] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2: stopping tablet replica
I20260321 02:58:15.741864 10068 raft_consensus.cc:2243] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 3 LEADER]: Raft consensus shutting down.
I20260321 02:58:15.742574 10068 raft_consensus.cc:2272] T f35df046e02643dab408161cee9f00a4 P 153c2de7193c4844b765382a198e9db2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.775311 10068 tablet_server.cc:196] TabletServer@127.9.213.1:0 shutdown complete.
I20260321 02:58:15.797796 10068 tablet_server.cc:179] TabletServer@127.9.213.2:0 shutting down...
I20260321 02:58:15.826114 10068 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 02:58:15.826956 10068 tablet_replica.cc:333] T 082582ef8a7a483eaf98b4ced6214a57 P 9f26c8327bfe4a58ae8b67940c9f49f8: stopping tablet replica
I20260321 02:58:15.827639 10068 raft_consensus.cc:2243] T 082582ef8a7a483eaf98b4ced6214a57 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:15.828166 10068 raft_consensus.cc:2272] T 082582ef8a7a483eaf98b4ced6214a57 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.829995 10068 tablet_replica.cc:333] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8: stopping tablet replica
I20260321 02:58:15.830732 10068 raft_consensus.cc:2243] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:15.831193 10068 raft_consensus.cc:2272] T b2633115df76454eb4c74d33d2b0b8fe P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.832944 10068 tablet_replica.cc:333] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8: stopping tablet replica
I20260321 02:58:15.833448 10068 raft_consensus.cc:2243] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:15.833877 10068 raft_consensus.cc:2272] T a56d5d5429b743bc8d2cb6fcb999f86e P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.835564 10068 tablet_replica.cc:333] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8: stopping tablet replica
I20260321 02:58:15.836143 10068 raft_consensus.cc:2243] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:15.836745 10068 raft_consensus.cc:2272] T 08f9bfdb4a464783b9b5b88025e96c4b P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.838595 10068 tablet_replica.cc:333] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8: stopping tablet replica
I20260321 02:58:15.839213 10068 raft_consensus.cc:2243] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 LEADER]: Raft consensus shutting down.
I20260321 02:58:15.840059 10068 raft_consensus.cc:2272] T 3a1c86bd318d452e9446f7ab8783c20f P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.841836 10068 tablet_replica.cc:333] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8: stopping tablet replica
I20260321 02:58:15.842381 10068 raft_consensus.cc:2243] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 LEADER]: Raft consensus shutting down.
I20260321 02:58:15.843202 10068 raft_consensus.cc:2272] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.845408 10068 tablet_replica.cc:333] T 60d9453110c241f4aaa3740c3a43e6b6 P 9f26c8327bfe4a58ae8b67940c9f49f8: stopping tablet replica
I20260321 02:58:15.846062 10068 raft_consensus.cc:2243] T 60d9453110c241f4aaa3740c3a43e6b6 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:15.846736 10068 raft_consensus.cc:2272] T 60d9453110c241f4aaa3740c3a43e6b6 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.848945 10068 tablet_replica.cc:333] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8: stopping tablet replica
W20260321 02:58:15.848861 10845 consensus_peers.cc:597] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e -> Peer 153c2de7193c4844b765382a198e9db2 (127.9.213.1:42123): Couldn't send request to peer 153c2de7193c4844b765382a198e9db2. Status: Network error: Client connection negotiation failed: client connection to 127.9.213.1:42123: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:15.849678 10068 raft_consensus.cc:2243] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 4 LEADER]: Raft consensus shutting down.
I20260321 02:58:15.850348 10068 raft_consensus.cc:2272] T e1d08251e74b419599f24a910bc9e538 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.851903 10068 tablet_replica.cc:333] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8: stopping tablet replica
I20260321 02:58:15.852425 10068 raft_consensus.cc:2243] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:15.852802 10068 raft_consensus.cc:2272] T 6e161a85964742b09837e0ae3dda87ca P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.854449 10068 tablet_replica.cc:333] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8: stopping tablet replica
I20260321 02:58:15.854849 10068 raft_consensus.cc:2243] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:15.855175 10068 raft_consensus.cc:2272] T f35df046e02643dab408161cee9f00a4 P 9f26c8327bfe4a58ae8b67940c9f49f8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.885541 10068 tablet_server.cc:196] TabletServer@127.9.213.2:0 shutdown complete.
I20260321 02:58:15.905427 10068 tablet_server.cc:179] TabletServer@127.9.213.3:0 shutting down...
I20260321 02:58:15.936950 10068 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260321 02:58:15.937776 10068 tablet_replica.cc:333] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e: stopping tablet replica
I20260321 02:58:15.938361 10068 raft_consensus.cc:2243] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:15.938817 10068 raft_consensus.cc:2272] T f35df046e02643dab408161cee9f00a4 P 5ed785fa1622433e8993e90b8c6c984e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.940711 10068 tablet_replica.cc:333] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e: stopping tablet replica
I20260321 02:58:15.941226 10068 raft_consensus.cc:2243] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [term 2 LEADER]: Raft consensus shutting down.
I20260321 02:58:15.941932 10068 raft_consensus.cc:2272] T 082582ef8a7a483eaf98b4ced6214a57 P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.943495 10068 tablet_replica.cc:333] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e: stopping tablet replica
I20260321 02:58:15.944087 10068 raft_consensus.cc:2243] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e [term 2 LEADER]: Raft consensus shutting down.
I20260321 02:58:15.944774 10068 raft_consensus.cc:2272] T 60d9453110c241f4aaa3740c3a43e6b6 P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.946334 10068 tablet_replica.cc:333] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e: stopping tablet replica
I20260321 02:58:15.946837 10068 raft_consensus.cc:2243] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 4 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:15.947247 10068 raft_consensus.cc:2272] T 6e161a85964742b09837e0ae3dda87ca P 5ed785fa1622433e8993e90b8c6c984e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.948940 10068 tablet_replica.cc:333] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e: stopping tablet replica
I20260321 02:58:15.949460 10068 raft_consensus.cc:2243] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 4 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:15.949887 10068 raft_consensus.cc:2272] T e1d08251e74b419599f24a910bc9e538 P 5ed785fa1622433e8993e90b8c6c984e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.964042 10068 tablet_replica.cc:333] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e: stopping tablet replica
I20260321 02:58:15.965029 10068 raft_consensus.cc:2243] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:15.965781 10068 raft_consensus.cc:2272] T 8619a3b4cc1b4d59b5a312ac0bc01017 P 5ed785fa1622433e8993e90b8c6c984e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.968348 10068 tablet_replica.cc:333] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e: stopping tablet replica
I20260321 02:58:15.969046 10068 raft_consensus.cc:2243] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:15.969671 10068 raft_consensus.cc:2272] T 3a1c86bd318d452e9446f7ab8783c20f P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.971750 10068 tablet_replica.cc:333] T 08f9bfdb4a464783b9b5b88025e96c4b P 5ed785fa1622433e8993e90b8c6c984e: stopping tablet replica
I20260321 02:58:15.972326 10068 raft_consensus.cc:2243] T 08f9bfdb4a464783b9b5b88025e96c4b P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:15.972936 10068 raft_consensus.cc:2272] T 08f9bfdb4a464783b9b5b88025e96c4b P 5ed785fa1622433e8993e90b8c6c984e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.974889 10068 tablet_replica.cc:333] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e: stopping tablet replica
I20260321 02:58:15.975366 10068 raft_consensus.cc:2243] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:15.976053 10068 raft_consensus.cc:2272] T a56d5d5429b743bc8d2cb6fcb999f86e P 5ed785fa1622433e8993e90b8c6c984e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:15.977911 10068 tablet_replica.cc:333] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e: stopping tablet replica
I20260321 02:58:15.978396 10068 raft_consensus.cc:2243] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 4 LEADER]: Raft consensus shutting down.
I20260321 02:58:15.979853 10068 raft_consensus.cc:2272] T b2633115df76454eb4c74d33d2b0b8fe P 5ed785fa1622433e8993e90b8c6c984e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:16.013980 10068 tablet_server.cc:196] TabletServer@127.9.213.3:0 shutdown complete.
I20260321 02:58:16.036979 10068 master.cc:562] Master@127.9.213.62:44005 shutting down...
I20260321 02:58:16.062963 10068 raft_consensus.cc:2243] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3 [term 1 LEADER]: Raft consensus shutting down.
I20260321 02:58:16.063720 10068 raft_consensus.cc:2272] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:16.064116 10068 tablet_replica.cc:333] T 00000000000000000000000000000000 P c392fd13682945a19bbbf698726015f3: stopping tablet replica
I20260321 02:58:16.084064 10068 master.cc:584] Master@127.9.213.62:44005 shutdown complete.
153c2de7193c4844b765382a198e9db2 5
5ed785fa1622433e8993e90b8c6c984e 1
9f26c8327bfe4a58ae8b67940c9f49f8 4
153c2de7193c4844b765382a198e9db2 3
5ed785fa1622433e8993e90b8c6c984e 3
9f26c8327bfe4a58ae8b67940c9f49f8 4
153c2de7193c4844b765382a198e9db2 8
5ed785fa1622433e8993e90b8c6c984e 1
9f26c8327bfe4a58ae8b67940c9f49f8 1
153c2de7193c4844b765382a198e9db2 4
5ed785fa1622433e8993e90b8c6c984e 3
9f26c8327bfe4a58ae8b67940c9f49f8 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28853 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260321 02:58:16.133122 10068 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.9.213.62:36307
I20260321 02:58:16.134824 10068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 02:58:16.142242 11109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 02:58:16.142787 11110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 02:58:16.143498 11112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 02:58:16.145231 10068 server_base.cc:1061] running on GCE node
I20260321 02:58:16.146219 10068 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 02:58:16.146523 10068 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 02:58:16.146740 10068 hybrid_clock.cc:648] HybridClock initialized: now 1774061896146697 us; error 0 us; skew 500 ppm
I20260321 02:58:16.147275 10068 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 02:58:16.150022 10068 webserver.cc:492] Webserver started at http://127.9.213.62:44769/ using document root <none> and password file <none>
I20260321 02:58:16.150552 10068 fs_manager.cc:362] Metadata directory not provided
I20260321 02:58:16.150780 10068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 02:58:16.151062 10068 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 02:58:16.152741 10068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/master-0-root/instance:
uuid: "c3992e3c917740db9e9d9c810eddf87c"
format_stamp: "Formatted at 2026-03-21 02:58:16 on dist-test-slave-mz2z"
I20260321 02:58:16.158670 10068 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260321 02:58:16.163717 11117 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 02:58:16.164856 10068 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260321 02:58:16.165428 10068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/master-0-root
uuid: "c3992e3c917740db9e9d9c810eddf87c"
format_stamp: "Formatted at 2026-03-21 02:58:16 on dist-test-slave-mz2z"
I20260321 02:58:16.165753 10068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 02:58:16.197937 10068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 02:58:16.199196 10068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 02:58:16.241067 10068 rpc_server.cc:307] RPC server started. Bound to: 127.9.213.62:36307
I20260321 02:58:16.241148 11168 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.213.62:36307 every 8 connection(s)
I20260321 02:58:16.245294 11169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:16.256624 11169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c: Bootstrap starting.
I20260321 02:58:16.263031 11169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:16.267755 11169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c: No bootstrap required, opened a new log
I20260321 02:58:16.270134 11169 raft_consensus.cc:359] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3992e3c917740db9e9d9c810eddf87c" member_type: VOTER }
I20260321 02:58:16.270596 11169 raft_consensus.cc:385] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:16.270980 11169 raft_consensus.cc:740] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3992e3c917740db9e9d9c810eddf87c, State: Initialized, Role: FOLLOWER
I20260321 02:58:16.271530 11169 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3992e3c917740db9e9d9c810eddf87c" member_type: VOTER }
I20260321 02:58:16.272150 11169 raft_consensus.cc:399] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260321 02:58:16.272456 11169 raft_consensus.cc:493] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260321 02:58:16.272732 11169 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:16.276911 11169 raft_consensus.cc:515] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3992e3c917740db9e9d9c810eddf87c" member_type: VOTER }
I20260321 02:58:16.277426 11169 leader_election.cc:304] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3992e3c917740db9e9d9c810eddf87c; no voters:
I20260321 02:58:16.278569 11169 leader_election.cc:290] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260321 02:58:16.279081 11172 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:16.280455 11172 raft_consensus.cc:697] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c [term 1 LEADER]: Becoming Leader. State: Replica: c3992e3c917740db9e9d9c810eddf87c, State: Running, Role: LEADER
I20260321 02:58:16.281188 11172 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3992e3c917740db9e9d9c810eddf87c" member_type: VOTER }
I20260321 02:58:16.281647 11169 sys_catalog.cc:565] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c [sys.catalog]: configured and running, proceeding with master startup.
I20260321 02:58:16.288470 11174 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3992e3c917740db9e9d9c810eddf87c. Latest consensus state: current_term: 1 leader_uuid: "c3992e3c917740db9e9d9c810eddf87c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3992e3c917740db9e9d9c810eddf87c" member_type: VOTER } }
I20260321 02:58:16.288967 11174 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c [sys.catalog]: This master's current role is: LEADER
I20260321 02:58:16.288903 11173 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3992e3c917740db9e9d9c810eddf87c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3992e3c917740db9e9d9c810eddf87c" member_type: VOTER } }
I20260321 02:58:16.289866 11173 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c [sys.catalog]: This master's current role is: LEADER
I20260321 02:58:16.291600 11181 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260321 02:58:16.300284 11181 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260321 02:58:16.301306 10068 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260321 02:58:16.310105 11181 catalog_manager.cc:1357] Generated new cluster ID: fc755aee32b64097a942ea251c0e6810
I20260321 02:58:16.310428 11181 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260321 02:58:16.328059 11181 catalog_manager.cc:1380] Generated new certificate authority record
I20260321 02:58:16.329649 11181 catalog_manager.cc:1514] Loading token signing keys...
I20260321 02:58:16.346614 11181 catalog_manager.cc:6044] T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c: Generated new TSK 0
I20260321 02:58:16.347288 11181 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260321 02:58:16.368643 10068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 02:58:16.375551 11190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 02:58:16.377233 11191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 02:58:16.379293 10068 server_base.cc:1061] running on GCE node
W20260321 02:58:16.379434 11193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 02:58:16.380743 10068 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 02:58:16.380944 10068 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 02:58:16.381062 10068 hybrid_clock.cc:648] HybridClock initialized: now 1774061896381051 us; error 0 us; skew 500 ppm
I20260321 02:58:16.381546 10068 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 02:58:16.384292 10068 webserver.cc:492] Webserver started at http://127.9.213.1:32851/ using document root <none> and password file <none>
I20260321 02:58:16.384793 10068 fs_manager.cc:362] Metadata directory not provided
I20260321 02:58:16.384958 10068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 02:58:16.385171 10068 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 02:58:16.386354 10068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-0-root/instance:
uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
format_stamp: "Formatted at 2026-03-21 02:58:16 on dist-test-slave-mz2z"
I20260321 02:58:16.391764 10068 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260321 02:58:16.396088 11198 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 02:58:16.397859 10068 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260321 02:58:16.398326 10068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-0-root
uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
format_stamp: "Formatted at 2026-03-21 02:58:16 on dist-test-slave-mz2z"
I20260321 02:58:16.398944 10068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 02:58:16.417981 10068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 02:58:16.419535 10068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 02:58:16.421622 10068 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 02:58:16.424229 10068 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 02:58:16.424486 10068 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 02:58:16.424767 10068 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 02:58:16.424958 10068 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 02:58:16.482530 10068 rpc_server.cc:307] RPC server started. Bound to: 127.9.213.1:36717
I20260321 02:58:16.482918 11260 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.213.1:36717 every 8 connection(s)
I20260321 02:58:16.488638 10068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 02:58:16.497694 11265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 02:58:16.500155 11266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 02:58:16.503782 11268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 02:58:16.503733 10068 server_base.cc:1061] running on GCE node
I20260321 02:58:16.506057 10068 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 02:58:16.506280 10068 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 02:58:16.506513 10068 hybrid_clock.cc:648] HybridClock initialized: now 1774061896506495 us; error 0 us; skew 500 ppm
I20260321 02:58:16.507325 10068 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 02:58:16.510911 10068 webserver.cc:492] Webserver started at http://127.9.213.2:35603/ using document root <none> and password file <none>
I20260321 02:58:16.511744 10068 fs_manager.cc:362] Metadata directory not provided
I20260321 02:58:16.511829 11261 heartbeater.cc:344] Connected to a master server at 127.9.213.62:36307
I20260321 02:58:16.512032 10068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 02:58:16.512565 11261 heartbeater.cc:461] Registering TS with master...
I20260321 02:58:16.512775 10068 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 02:58:16.513388 11261 heartbeater.cc:507] Master 127.9.213.62:36307 requested a full tablet report, sending...
I20260321 02:58:16.514227 10068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-1-root/instance:
uuid: "83b12b883a7f4129a87dadf5a78f08da"
format_stamp: "Formatted at 2026-03-21 02:58:16 on dist-test-slave-mz2z"
I20260321 02:58:16.516077 11134 ts_manager.cc:194] Registered new tserver with Master: 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717)
I20260321 02:58:16.519065 11134 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42502
I20260321 02:58:16.521011 10068 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260321 02:58:16.525118 11273 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 02:58:16.526114 10068 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260321 02:58:16.526508 10068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-1-root
uuid: "83b12b883a7f4129a87dadf5a78f08da"
format_stamp: "Formatted at 2026-03-21 02:58:16 on dist-test-slave-mz2z"
I20260321 02:58:16.526859 10068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 02:58:16.547048 10068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 02:58:16.548559 10068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 02:58:16.550311 10068 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 02:58:16.553398 10068 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 02:58:16.553686 10068 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 02:58:16.553982 10068 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 02:58:16.554188 10068 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 02:58:16.611888 10068 rpc_server.cc:307] RPC server started. Bound to: 127.9.213.2:34267
I20260321 02:58:16.612066 11335 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.213.2:34267 every 8 connection(s)
I20260321 02:58:16.619270 10068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 02:58:16.626956 11339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 02:58:16.628449 11340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 02:58:16.632316 10068 server_base.cc:1061] running on GCE node
W20260321 02:58:16.633953 11342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 02:58:16.635293 11336 heartbeater.cc:344] Connected to a master server at 127.9.213.62:36307
I20260321 02:58:16.635773 11336 heartbeater.cc:461] Registering TS with master...
I20260321 02:58:16.635807 10068 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 02:58:16.636652 10068 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 02:58:16.636976 11336 heartbeater.cc:507] Master 127.9.213.62:36307 requested a full tablet report, sending...
I20260321 02:58:16.637265 10068 hybrid_clock.cc:648] HybridClock initialized: now 1774061896637243 us; error 0 us; skew 500 ppm
I20260321 02:58:16.638216 10068 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 02:58:16.640467 11134 ts_manager.cc:194] Registered new tserver with Master: 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:16.643103 10068 webserver.cc:492] Webserver started at http://127.9.213.3:37895/ using document root <none> and password file <none>
I20260321 02:58:16.643723 11134 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42514
I20260321 02:58:16.644264 10068 fs_manager.cc:362] Metadata directory not provided
I20260321 02:58:16.644551 10068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 02:58:16.644877 10068 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 02:58:16.646184 10068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-2-root/instance:
uuid: "fef7f8cc55b043039131bd9b2229f0a5"
format_stamp: "Formatted at 2026-03-21 02:58:16 on dist-test-slave-mz2z"
I20260321 02:58:16.652420 10068 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260321 02:58:16.656893 11347 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 02:58:16.657992 10068 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20260321 02:58:16.658370 10068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-2-root
uuid: "fef7f8cc55b043039131bd9b2229f0a5"
format_stamp: "Formatted at 2026-03-21 02:58:16 on dist-test-slave-mz2z"
I20260321 02:58:16.658741 10068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 02:58:16.676964 10068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 02:58:16.678668 10068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 02:58:16.680279 10068 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 02:58:16.683120 10068 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 02:58:16.683547 10068 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260321 02:58:16.683895 10068 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 02:58:16.684491 10068 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 02:58:16.736191 10068 rpc_server.cc:307] RPC server started. Bound to: 127.9.213.3:43533
I20260321 02:58:16.736274 11409 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.213.3:43533 every 8 connection(s)
I20260321 02:58:16.755324 11410 heartbeater.cc:344] Connected to a master server at 127.9.213.62:36307
I20260321 02:58:16.755705 11410 heartbeater.cc:461] Registering TS with master...
I20260321 02:58:16.756706 11410 heartbeater.cc:507] Master 127.9.213.62:36307 requested a full tablet report, sending...
I20260321 02:58:16.759048 11134 ts_manager.cc:194] Registered new tserver with Master: fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:16.759464 10068 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018494803s
I20260321 02:58:16.760677 11134 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42530
I20260321 02:58:16.790544 11134 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42544:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260321 02:58:16.797289 11134 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260321 02:58:17.007875 11375 tablet_service.cc:1511] Processing CreateTablet for tablet aab7e035f7984ed8ba6560e1591ec831 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260321 02:58:17.008852 11374 tablet_service.cc:1511] Processing CreateTablet for tablet 8ac864becf784ae392582f636a51863e (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260321 02:58:17.009487 11375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aab7e035f7984ed8ba6560e1591ec831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.010288 11374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ac864becf784ae392582f636a51863e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.010344 11372 tablet_service.cc:1511] Processing CreateTablet for tablet d42ec1d798814a50bb535a7012a76eb4 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260321 02:58:17.011866 11372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d42ec1d798814a50bb535a7012a76eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.012717 11373 tablet_service.cc:1511] Processing CreateTablet for tablet 70e1ea01442f4d13a0f6432ea463c5df (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260321 02:58:17.014047 11373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e1ea01442f4d13a0f6432ea463c5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.019430 11225 tablet_service.cc:1511] Processing CreateTablet for tablet 8ac864becf784ae392582f636a51863e (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260321 02:58:17.020298 11226 tablet_service.cc:1511] Processing CreateTablet for tablet aab7e035f7984ed8ba6560e1591ec831 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260321 02:58:17.020889 11225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ac864becf784ae392582f636a51863e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.021618 11226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aab7e035f7984ed8ba6560e1591ec831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.022997 11224 tablet_service.cc:1511] Processing CreateTablet for tablet 70e1ea01442f4d13a0f6432ea463c5df (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260321 02:58:17.024402 11224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e1ea01442f4d13a0f6432ea463c5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.029273 11223 tablet_service.cc:1511] Processing CreateTablet for tablet d42ec1d798814a50bb535a7012a76eb4 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260321 02:58:17.030822 11223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d42ec1d798814a50bb535a7012a76eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.033159 11222 tablet_service.cc:1511] Processing CreateTablet for tablet 562af69310cc422e8b378a98d1bb279b (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260321 02:58:17.034535 11222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 562af69310cc422e8b378a98d1bb279b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.040123 11371 tablet_service.cc:1511] Processing CreateTablet for tablet 562af69310cc422e8b378a98d1bb279b (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260321 02:58:17.041646 11371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 562af69310cc422e8b378a98d1bb279b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.044466 11221 tablet_service.cc:1511] Processing CreateTablet for tablet c5fc517dbbd646bfa0d307d4e42556ba (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260321 02:58:17.045972 11221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5fc517dbbd646bfa0d307d4e42556ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.057633 11370 tablet_service.cc:1511] Processing CreateTablet for tablet c5fc517dbbd646bfa0d307d4e42556ba (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260321 02:58:17.059162 11370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5fc517dbbd646bfa0d307d4e42556ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.064997 11220 tablet_service.cc:1511] Processing CreateTablet for tablet 5aaa96ba673046439e87b58b0e693526 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260321 02:58:17.066553 11220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aaa96ba673046439e87b58b0e693526. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.106055 11373 tablet_service.cc:1511] Processing CreateTablet for tablet 8f4ec8d11b7b4e6096d8c2b8ece078dc (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260321 02:58:17.106199 11219 tablet_service.cc:1511] Processing CreateTablet for tablet 8f4ec8d11b7b4e6096d8c2b8ece078dc (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260321 02:58:17.107496 11373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f4ec8d11b7b4e6096d8c2b8ece078dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.107604 11219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f4ec8d11b7b4e6096d8c2b8ece078dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.109262 11218 tablet_service.cc:1511] Processing CreateTablet for tablet 7d5b2547bde048c88e963f650d905f76 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260321 02:58:17.110706 11218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d5b2547bde048c88e963f650d905f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.110798 11369 tablet_service.cc:1511] Processing CreateTablet for tablet 7d5b2547bde048c88e963f650d905f76 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260321 02:58:17.113260 11371 tablet_service.cc:1511] Processing CreateTablet for tablet bd04df7a33b14699ab2d2bd6325942e4 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260321 02:58:17.103263 11223 tablet_service.cc:1511] Processing CreateTablet for tablet bd04df7a33b14699ab2d2bd6325942e4 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260321 02:58:17.112677 11369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d5b2547bde048c88e963f650d905f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.118091 11217 tablet_service.cc:1511] Processing CreateTablet for tablet 6f453202f8fb47abaa1cc6c0fc390553 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260321 02:58:17.119735 11217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f453202f8fb47abaa1cc6c0fc390553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.103291 11424 tablet_bootstrap.cc:492] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:17.124358 11371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd04df7a33b14699ab2d2bd6325942e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.124742 11223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd04df7a33b14699ab2d2bd6325942e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.145264 11425 tablet_bootstrap.cc:492] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:17.132378 11372 tablet_service.cc:1511] Processing CreateTablet for tablet 5aaa96ba673046439e87b58b0e693526 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260321 02:58:17.152354 11372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aaa96ba673046439e87b58b0e693526. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.158252 11374 tablet_service.cc:1511] Processing CreateTablet for tablet 6f453202f8fb47abaa1cc6c0fc390553 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260321 02:58:17.159621 11374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f453202f8fb47abaa1cc6c0fc390553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.179648 11424 tablet_bootstrap.cc:654] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:17.181155 11425 tablet_bootstrap.cc:654] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:17.190443 11223 tablet_service.cc:1511] Processing CreateTablet for tablet 72e86c671bb747c785d8f676ed3b8315 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260321 02:58:17.200826 11223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72e86c671bb747c785d8f676ed3b8315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.204861 11301 tablet_service.cc:1511] Processing CreateTablet for tablet aab7e035f7984ed8ba6560e1591ec831 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260321 02:58:17.206547 11301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aab7e035f7984ed8ba6560e1591ec831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.210638 11300 tablet_service.cc:1511] Processing CreateTablet for tablet 8ac864becf784ae392582f636a51863e (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260321 02:58:17.213869 11299 tablet_service.cc:1511] Processing CreateTablet for tablet 70e1ea01442f4d13a0f6432ea463c5df (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260321 02:58:17.216717 11298 tablet_service.cc:1511] Processing CreateTablet for tablet d42ec1d798814a50bb535a7012a76eb4 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260321 02:58:17.219502 11297 tablet_service.cc:1511] Processing CreateTablet for tablet 562af69310cc422e8b378a98d1bb279b (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260321 02:58:17.223438 11296 tablet_service.cc:1511] Processing CreateTablet for tablet c5fc517dbbd646bfa0d307d4e42556ba (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260321 02:58:17.224856 11223 tablet_service.cc:1511] Processing CreateTablet for tablet 404c3f80af814c3fa8db2501818b1e33 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260321 02:58:17.225855 11372 tablet_service.cc:1511] Processing CreateTablet for tablet e8b6fd0841c644779c26179d18ddf697 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260321 02:58:17.227702 11372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8b6fd0841c644779c26179d18ddf697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.226199 11223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 404c3f80af814c3fa8db2501818b1e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.227857 11295 tablet_service.cc:1511] Processing CreateTablet for tablet 5aaa96ba673046439e87b58b0e693526 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260321 02:58:17.192855 11369 tablet_service.cc:1511] Processing CreateTablet for tablet 72e86c671bb747c785d8f676ed3b8315 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260321 02:58:17.231177 11375 tablet_service.cc:1511] Processing CreateTablet for tablet 404c3f80af814c3fa8db2501818b1e33 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260321 02:58:17.231140 11294 tablet_service.cc:1511] Processing CreateTablet for tablet bd04df7a33b14699ab2d2bd6325942e4 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260321 02:58:17.234130 11221 tablet_service.cc:1511] Processing CreateTablet for tablet b968fa0c5fb84474b7445627011bf12a (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260321 02:58:17.239995 11221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b968fa0c5fb84474b7445627011bf12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.241199 11217 tablet_service.cc:1511] Processing CreateTablet for tablet 0225614ab358492abd584946a97a42f2 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260321 02:58:17.242542 11217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0225614ab358492abd584946a97a42f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.235409 11371 tablet_service.cc:1511] Processing CreateTablet for tablet b968fa0c5fb84474b7445627011bf12a (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260321 02:58:17.231838 11300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ac864becf784ae392582f636a51863e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.239065 11369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72e86c671bb747c785d8f676ed3b8315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.242205 11374 tablet_service.cc:1511] Processing CreateTablet for tablet 0225614ab358492abd584946a97a42f2 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260321 02:58:17.244920 11374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0225614ab358492abd584946a97a42f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.245669 11375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 404c3f80af814c3fa8db2501818b1e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.246302 11371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b968fa0c5fb84474b7445627011bf12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.246922 11294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd04df7a33b14699ab2d2bd6325942e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.252414 11299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e1ea01442f4d13a0f6432ea463c5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.256338 11298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d42ec1d798814a50bb535a7012a76eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.260392 11296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5fc517dbbd646bfa0d307d4e42556ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.264513 11295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aaa96ba673046439e87b58b0e693526. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.268129 11297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 562af69310cc422e8b378a98d1bb279b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.269201 11219 tablet_service.cc:1511] Processing CreateTablet for tablet e8b6fd0841c644779c26179d18ddf697 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260321 02:58:17.270478 11219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8b6fd0841c644779c26179d18ddf697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.231194 11370 tablet_service.cc:1511] Processing CreateTablet for tablet 709aaa09afd74a25bbd5dc62dca37f02 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260321 02:58:17.194983 11220 tablet_service.cc:1511] Processing CreateTablet for tablet 709aaa09afd74a25bbd5dc62dca37f02 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260321 02:58:17.316105 11370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 709aaa09afd74a25bbd5dc62dca37f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.319252 11293 tablet_service.cc:1511] Processing CreateTablet for tablet 8f4ec8d11b7b4e6096d8c2b8ece078dc (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260321 02:58:17.320785 11293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f4ec8d11b7b4e6096d8c2b8ece078dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.326763 11220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 709aaa09afd74a25bbd5dc62dca37f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.341070 11424 tablet_bootstrap.cc:492] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:17.342110 11424 ts_tablet_manager.cc:1403] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.239s user 0.017s sys 0.026s
I20260321 02:58:17.341993 11292 tablet_service.cc:1511] Processing CreateTablet for tablet 7d5b2547bde048c88e963f650d905f76 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260321 02:58:17.345822 11424 raft_consensus.cc:359] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:17.365226 11182 auto_rebalancer.cc:247] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260321 02:58:17.368605 11292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d5b2547bde048c88e963f650d905f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.374512 11425 tablet_bootstrap.cc:492] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:17.376310 11425 ts_tablet_manager.cc:1403] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.231s user 0.054s sys 0.074s
I20260321 02:58:17.379261 11425 raft_consensus.cc:359] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } }
I20260321 02:58:17.362221 11424 raft_consensus.cc:385] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:17.384258 11424 raft_consensus.cc:740] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:17.385211 11424 consensus_queue.cc:260] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:17.386726 11425 raft_consensus.cc:385] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:17.387121 11425 raft_consensus.cc:740] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
W20260321 02:58:17.392082 11119 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 67777.9us
I20260321 02:58:17.396075 11425 consensus_queue.cc:260] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } }
I20260321 02:58:17.396817 11374 tablet_service.cc:1511] Processing CreateTablet for tablet ee109699b22b4cd08d4e2ac0ba6f3239 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260321 02:58:17.397374 11223 tablet_service.cc:1511] Processing CreateTablet for tablet 515945e5685c4afe9d41d5317acb434c (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260321 02:58:17.401176 11374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee109699b22b4cd08d4e2ac0ba6f3239. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.401628 11223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 515945e5685c4afe9d41d5317acb434c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.401651 11299 tablet_service.cc:1511] Processing CreateTablet for tablet 6f453202f8fb47abaa1cc6c0fc390553 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260321 02:58:17.397770 11375 tablet_service.cc:1511] Processing CreateTablet for tablet 515945e5685c4afe9d41d5317acb434c (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260321 02:58:17.408978 11375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 515945e5685c4afe9d41d5317acb434c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.406826 11295 tablet_service.cc:1511] Processing CreateTablet for tablet 72e86c671bb747c785d8f676ed3b8315 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260321 02:58:17.421373 11424 ts_tablet_manager.cc:1434] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.079s user 0.009s sys 0.001s
I20260321 02:58:17.421373 11217 tablet_service.cc:1511] Processing CreateTablet for tablet 48b15e4599b848c8a833e37849eabbda (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260321 02:58:17.412714 11219 tablet_service.cc:1511] Processing CreateTablet for tablet ee109699b22b4cd08d4e2ac0ba6f3239 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260321 02:58:17.422734 11410 heartbeater.cc:499] Master 127.9.213.62:36307 was elected leader, sending a full tablet report...
I20260321 02:58:17.422863 11217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48b15e4599b848c8a833e37849eabbda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.430109 11219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee109699b22b4cd08d4e2ac0ba6f3239. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.440567 11299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f453202f8fb47abaa1cc6c0fc390553. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.444766 11295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72e86c671bb747c785d8f676ed3b8315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.466044 11219 tablet_service.cc:1511] Processing CreateTablet for tablet b10f30fc9cc94383afab2980647cbb2e (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260321 02:58:17.466917 11369 tablet_service.cc:1511] Processing CreateTablet for tablet 48b15e4599b848c8a833e37849eabbda (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260321 02:58:17.467558 11219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b10f30fc9cc94383afab2980647cbb2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.467227 11371 tablet_service.cc:1511] Processing CreateTablet for tablet b10f30fc9cc94383afab2980647cbb2e (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260321 02:58:17.468315 11369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48b15e4599b848c8a833e37849eabbda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.468911 11371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b10f30fc9cc94383afab2980647cbb2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.469899 11424 tablet_bootstrap.cc:492] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:17.482126 11424 tablet_bootstrap.cc:654] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:17.482419 11429 tablet_bootstrap.cc:492] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:17.483222 11425 ts_tablet_manager.cc:1434] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.106s user 0.015s sys 0.006s
I20260321 02:58:17.484057 11261 heartbeater.cc:499] Master 127.9.213.62:36307 was elected leader, sending a full tablet report...
I20260321 02:58:17.494983 11221 tablet_service.cc:1511] Processing CreateTablet for tablet 36e68ffcc96a423c8e07ac7cbfaa149a (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260321 02:58:17.496508 11293 tablet_service.cc:1511] Processing CreateTablet for tablet 404c3f80af814c3fa8db2501818b1e33 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260321 02:58:17.497324 11221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e68ffcc96a423c8e07ac7cbfaa149a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.497812 11293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 404c3f80af814c3fa8db2501818b1e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.500316 11425 tablet_bootstrap.cc:492] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:17.460371 11294 tablet_service.cc:1511] Processing CreateTablet for tablet 709aaa09afd74a25bbd5dc62dca37f02 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260321 02:58:17.506841 11425 tablet_bootstrap.cc:654] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:17.508177 11294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 709aaa09afd74a25bbd5dc62dca37f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.509725 11298 tablet_service.cc:1511] Processing CreateTablet for tablet 0225614ab358492abd584946a97a42f2 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260321 02:58:17.511006 11298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0225614ab358492abd584946a97a42f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.515385 11292 tablet_service.cc:1511] Processing CreateTablet for tablet 515945e5685c4afe9d41d5317acb434c (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260321 02:58:17.517113 11428 raft_consensus.cc:493] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:17.476526 11297 tablet_service.cc:1511] Processing CreateTablet for tablet e8b6fd0841c644779c26179d18ddf697 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260321 02:58:17.539662 11428 raft_consensus.cc:515] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:17.540565 11430 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:17.540784 11297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8b6fd0841c644779c26179d18ddf697. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.541064 11430 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } }
I20260321 02:58:17.546866 11292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 515945e5685c4afe9d41d5317acb434c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.486203 11296 tablet_service.cc:1511] Processing CreateTablet for tablet b968fa0c5fb84474b7445627011bf12a (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260321 02:58:17.550076 11296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b968fa0c5fb84474b7445627011bf12a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.558231 11428 leader_election.cc:290] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:17.561968 11424 tablet_bootstrap.cc:492] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:17.562497 11429 tablet_bootstrap.cc:654] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:17.562569 11424 ts_tablet_manager.cc:1403] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.093s user 0.016s sys 0.004s
I20260321 02:58:17.569404 11371 tablet_service.cc:1511] Processing CreateTablet for tablet 36e68ffcc96a423c8e07ac7cbfaa149a (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260321 02:58:17.569984 11224 tablet_service.cc:1511] Processing CreateTablet for tablet 513757b127054285b0686cd45044c596 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260321 02:58:17.572887 11224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 513757b127054285b0686cd45044c596. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.573167 11218 tablet_service.cc:1511] Processing CreateTablet for tablet 4be17edc893b4dea9368c0e67b931ef6 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260321 02:58:17.574421 11218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4be17edc893b4dea9368c0e67b931ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.570013 11424 raft_consensus.cc:359] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:17.576282 11424 raft_consensus.cc:385] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:17.576623 11424 raft_consensus.cc:740] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:17.586964 11371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e68ffcc96a423c8e07ac7cbfaa149a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.591325 11301 tablet_service.cc:1511] Processing CreateTablet for tablet ee109699b22b4cd08d4e2ac0ba6f3239 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260321 02:58:17.592720 11301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee109699b22b4cd08d4e2ac0ba6f3239. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.601537 11217 tablet_service.cc:1511] Processing CreateTablet for tablet 467887dd1b5b410390d81755f38f0deb (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260321 02:58:17.601761 11372 tablet_service.cc:1511] Processing CreateTablet for tablet 4be17edc893b4dea9368c0e67b931ef6 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260321 02:58:17.602025 11374 tablet_service.cc:1511] Processing CreateTablet for tablet 513757b127054285b0686cd45044c596 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260321 02:58:17.603150 11372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4be17edc893b4dea9368c0e67b931ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.603717 11217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 467887dd1b5b410390d81755f38f0deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.604559 11374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 513757b127054285b0686cd45044c596. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.608199 11430 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:17.608372 11226 tablet_service.cc:1511] Processing CreateTablet for tablet e4337508daeb4d598c4f6a0b4314878d (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260321 02:58:17.609802 11226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4337508daeb4d598c4f6a0b4314878d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.609354 11222 tablet_service.cc:1511] Processing CreateTablet for tablet 1146889bd4dc49e9a2b67700208bc264 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260321 02:58:17.610872 11222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1146889bd4dc49e9a2b67700208bc264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.615279 11424 consensus_queue.cc:260] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:17.615805 11375 tablet_service.cc:1511] Processing CreateTablet for tablet 467887dd1b5b410390d81755f38f0deb (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260321 02:58:17.617383 11375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 467887dd1b5b410390d81755f38f0deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.615892 11373 tablet_service.cc:1511] Processing CreateTablet for tablet e4337508daeb4d598c4f6a0b4314878d (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260321 02:58:17.619692 11373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4337508daeb4d598c4f6a0b4314878d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.645246 11293 tablet_service.cc:1511] Processing CreateTablet for tablet 48b15e4599b848c8a833e37849eabbda (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260321 02:58:17.646924 11336 heartbeater.cc:499] Master 127.9.213.62:36307 was elected leader, sending a full tablet report...
I20260321 02:58:17.686980 11300 tablet_service.cc:1511] Processing CreateTablet for tablet b10f30fc9cc94383afab2980647cbb2e (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260321 02:58:17.696573 11220 tablet_service.cc:1511] Processing CreateTablet for tablet 6e21cb03b1624c9fa5acfa1cd5d5db8f (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260321 02:58:17.699177 11225 tablet_service.cc:1511] Processing CreateTablet for tablet c104d8b810534a6c918eab2f919a83dc (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260321 02:58:17.709117 11370 tablet_service.cc:1511] Processing CreateTablet for tablet 1146889bd4dc49e9a2b67700208bc264 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260321 02:58:17.710884 11370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1146889bd4dc49e9a2b67700208bc264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.716614 11424 ts_tablet_manager.cc:1434] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.154s user 0.013s sys 0.004s
I20260321 02:58:17.717803 11424 tablet_bootstrap.cc:492] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:17.717999 11293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48b15e4599b848c8a833e37849eabbda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.721601 11223 tablet_service.cc:1511] Processing CreateTablet for tablet f9e8870ff9214a65af6cd9f677bffab4 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260321 02:58:17.723465 11218 tablet_service.cc:1511] Processing CreateTablet for tablet 56f96ea607c24b239b4eb847bdf1e142 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260321 02:58:17.725171 11368 tablet_service.cc:1511] Processing CreateTablet for tablet 56f96ea607c24b239b4eb847bdf1e142 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260321 02:58:17.725843 11366 tablet_service.cc:1511] Processing CreateTablet for tablet c104d8b810534a6c918eab2f919a83dc (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260321 02:58:17.726688 11368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56f96ea607c24b239b4eb847bdf1e142. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.744486 11220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e21cb03b1624c9fa5acfa1cd5d5db8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.745201 11367 tablet_service.cc:1511] Processing CreateTablet for tablet 6e21cb03b1624c9fa5acfa1cd5d5db8f (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260321 02:58:17.746723 11367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e21cb03b1624c9fa5acfa1cd5d5db8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.748919 11366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c104d8b810534a6c918eab2f919a83dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.749953 11300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b10f30fc9cc94383afab2980647cbb2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.751139 11297 tablet_service.cc:1511] Processing CreateTablet for tablet 36e68ffcc96a423c8e07ac7cbfaa149a (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260321 02:58:17.772518 11297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36e68ffcc96a423c8e07ac7cbfaa149a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.775331 11425 tablet_bootstrap.cc:492] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:17.775929 11425 ts_tablet_manager.cc:1403] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.276s user 0.052s sys 0.036s
I20260321 02:58:17.779109 11425 raft_consensus.cc:359] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:17.779919 11425 raft_consensus.cc:385] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:17.784056 11425 raft_consensus.cc:740] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:17.784982 11425 consensus_queue.cc:260] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:17.799095 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:17.802067 11225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c104d8b810534a6c918eab2f919a83dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.803164 11221 tablet_service.cc:1511] Processing CreateTablet for tablet 2fcb8dea530d4005a0101183a89b709d (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260321 02:58:17.804509 11221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fcb8dea530d4005a0101183a89b709d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.804872 11428 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:17.805667 11428 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:17.808529 11223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9e8870ff9214a65af6cd9f677bffab4. 1 dirs total, 0 dirs full, 0 dirs failed
W20260321 02:58:17.809116 11200 leader_election.cc:343] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:17.807746 11428 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:17.813652 11219 tablet_service.cc:1511] Processing CreateTablet for tablet ca80e3e3473540618f654536751cdc3b (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260321 02:58:17.815002 11219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca80e3e3473540618f654536751cdc3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.820133 11218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56f96ea607c24b239b4eb847bdf1e142. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.830139 11374 tablet_service.cc:1511] Processing CreateTablet for tablet f9e8870ff9214a65af6cd9f677bffab4 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260321 02:58:17.831542 11374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9e8870ff9214a65af6cd9f677bffab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.817117 11424 tablet_bootstrap.cc:654] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:17.836753 11430 raft_consensus.cc:493] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:17.837276 11430 raft_consensus.cc:515] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:17.839178 11425 ts_tablet_manager.cc:1434] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.063s user 0.001s sys 0.005s
I20260321 02:58:17.839323 11430 leader_election.cc:290] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:17.840376 11425 tablet_bootstrap.cc:492] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:17.846580 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:17.848928 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:17.850481 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:58:17.850585 11349 leader_election.cc:343] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:17.851161 11385 raft_consensus.cc:2468] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
W20260321 02:58:17.851542 11349 leader_election.cc:343] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:17.852958 11200 leader_election.cc:304] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:17.854847 11430 raft_consensus.cc:2804] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:17.855232 11430 raft_consensus.cc:493] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:17.855574 11430 raft_consensus.cc:3060] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:17.860531 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:17.866195 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:17.868456 11429 tablet_bootstrap.cc:492] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:17.868961 11429 ts_tablet_manager.cc:1403] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.387s user 0.024s sys 0.030s
W20260321 02:58:17.874704 11349 leader_election.cc:343] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:17.876523 11349 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:17.863322 11299 tablet_service.cc:1511] Processing CreateTablet for tablet 4be17edc893b4dea9368c0e67b931ef6 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260321 02:58:17.884855 11294 tablet_service.cc:1511] Processing CreateTablet for tablet 513757b127054285b0686cd45044c596 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260321 02:58:17.886430 11294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 513757b127054285b0686cd45044c596. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.878844 11428 raft_consensus.cc:2749] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:17.879031 11429 raft_consensus.cc:359] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:17.890357 11429 raft_consensus.cc:385] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:17.890681 11429 raft_consensus.cc:740] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:17.894120 11296 tablet_service.cc:1511] Processing CreateTablet for tablet 467887dd1b5b410390d81755f38f0deb (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260321 02:58:17.887841 11430 raft_consensus.cc:515] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:17.930189 11295 tablet_service.cc:1511] Processing CreateTablet for tablet e4337508daeb4d598c4f6a0b4314878d (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260321 02:58:17.930579 11296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 467887dd1b5b410390d81755f38f0deb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.897732 11372 tablet_service.cc:1511] Processing CreateTablet for tablet 2fcb8dea530d4005a0101183a89b709d (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260321 02:58:17.901928 11224 tablet_service.cc:1511] Processing CreateTablet for tablet 76097e48b77445aaa1ffe490b8bb41cf (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260321 02:58:17.906668 11294 tablet_service.cc:1511] Processing CreateTablet for tablet c104d8b810534a6c918eab2f919a83dc (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260321 02:58:17.931627 11295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4337508daeb4d598c4f6a0b4314878d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.931689 11372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fcb8dea530d4005a0101183a89b709d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.906669 11301 tablet_service.cc:1511] Processing CreateTablet for tablet f9e8870ff9214a65af6cd9f677bffab4 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260321 02:58:17.908025 11369 tablet_service.cc:1511] Processing CreateTablet for tablet 76097e48b77445aaa1ffe490b8bb41cf (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260321 02:58:17.891543 11429 consensus_queue.cc:260] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:17.934774 11368 tablet_service.cc:1511] Processing CreateTablet for tablet 3b4e9b11d3c54f328ebeeee4854be727 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260321 02:58:17.901247 11371 tablet_service.cc:1511] Processing CreateTablet for tablet ca80e3e3473540618f654536751cdc3b (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260321 02:58:17.936713 11371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca80e3e3473540618f654536751cdc3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.938445 11368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b4e9b11d3c54f328ebeeee4854be727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.920648 11298 tablet_service.cc:1511] Processing CreateTablet for tablet 1146889bd4dc49e9a2b67700208bc264 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260321 02:58:17.938817 11224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76097e48b77445aaa1ffe490b8bb41cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.928858 11297 tablet_service.cc:1511] Processing CreateTablet for tablet 56f96ea607c24b239b4eb847bdf1e142 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260321 02:58:17.930646 11236 raft_consensus.cc:2393] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 1: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:17.943849 11369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76097e48b77445aaa1ffe490b8bb41cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.952471 11349 leader_election.cc:304] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:17.952194 11424 tablet_bootstrap.cc:492] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:17.953840 11428 raft_consensus.cc:3060] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:17.956161 11430 leader_election.cc:290] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:17.958047 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:17.960421 11218 tablet_service.cc:1511] Processing CreateTablet for tablet 42855a31af244db9b41f42c30304de34 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260321 02:58:17.926292 11222 tablet_service.cc:1511] Processing CreateTablet for tablet 8a38bdd851a24a9a9c5e835ac457cda2 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260321 02:58:17.954253 11367 tablet_service.cc:1511] Processing CreateTablet for tablet 006e9dd81c0c4368bed549c97128f4ea (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260321 02:58:17.968667 11367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 006e9dd81c0c4368bed549c97128f4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.968207 11366 tablet_service.cc:1511] Processing CreateTablet for tablet 42855a31af244db9b41f42c30304de34 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260321 02:58:17.969813 11366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42855a31af244db9b41f42c30304de34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.976063 11218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42855a31af244db9b41f42c30304de34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.976733 11299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4be17edc893b4dea9368c0e67b931ef6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.977411 11217 tablet_service.cc:1511] Processing CreateTablet for tablet 3b4e9b11d3c54f328ebeeee4854be727 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260321 02:58:17.967643 11425 tablet_bootstrap.cc:654] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:17.978861 11217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b4e9b11d3c54f328ebeeee4854be727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.980307 11297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56f96ea607c24b239b4eb847bdf1e142. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.982479 11222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a38bdd851a24a9a9c5e835ac457cda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.970463 11226 tablet_service.cc:1511] Processing CreateTablet for tablet 006e9dd81c0c4368bed549c97128f4ea (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260321 02:58:17.985635 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:17.988000 11220 tablet_service.cc:1511] Processing CreateTablet for tablet 62b43ed6f6404b7aa8463e168f98232a (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
W20260321 02:58:17.988953 11200 leader_election.cc:343] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:17.989187 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:17.989532 11200 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:58:17.990932 11200 leader_election.cc:343] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:17.992028 11430 raft_consensus.cc:2749] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:17.987663 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:58:17.994241 11200 leader_election.cc:343] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:17.983745 11226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 006e9dd81c0c4368bed549c97128f4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.996151 11301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9e8870ff9214a65af6cd9f677bffab4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.996623 11293 tablet_service.cc:1511] Processing CreateTablet for tablet 6e21cb03b1624c9fa5acfa1cd5d5db8f (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260321 02:58:17.998044 11293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e21cb03b1624c9fa5acfa1cd5d5db8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:17.999416 11373 tablet_service.cc:1511] Processing CreateTablet for tablet fd5fbcf5c6ce4a819d81ed0d22bf7636 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260321 02:58:18.005609 11373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd5fbcf5c6ce4a819d81ed0d22bf7636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.001186 11370 tablet_service.cc:1511] Processing CreateTablet for tablet 8a38bdd851a24a9a9c5e835ac457cda2 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260321 02:58:18.002751 11300 tablet_service.cc:1511] Processing CreateTablet for tablet 2fcb8dea530d4005a0101183a89b709d (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260321 02:58:18.002797 11221 tablet_service.cc:1511] Processing CreateTablet for tablet c2f6dd25ab2545e59d5922d37e1ec68d (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260321 02:58:18.006094 11292 tablet_service.cc:1511] Processing CreateTablet for tablet 76097e48b77445aaa1ffe490b8bb41cf (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260321 02:58:18.025565 11298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1146889bd4dc49e9a2b67700208bc264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.026335 11294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c104d8b810534a6c918eab2f919a83dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.016176 11219 tablet_service.cc:1511] Processing CreateTablet for tablet fd5fbcf5c6ce4a819d81ed0d22bf7636 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260321 02:58:18.040299 11220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62b43ed6f6404b7aa8463e168f98232a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.041111 11221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2f6dd25ab2545e59d5922d37e1ec68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.042097 11300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fcb8dea530d4005a0101183a89b709d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.043797 11299 tablet_service.cc:1511] Processing CreateTablet for tablet ca80e3e3473540618f654536751cdc3b (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260321 02:58:18.045070 11370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a38bdd851a24a9a9c5e835ac457cda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.047920 11219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd5fbcf5c6ce4a819d81ed0d22bf7636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.045162 11299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca80e3e3473540618f654536751cdc3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.060251 11292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76097e48b77445aaa1ffe490b8bb41cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.069833 11299 tablet_service.cc:1511] Processing CreateTablet for tablet fd5fbcf5c6ce4a819d81ed0d22bf7636 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260321 02:58:17.960441 11424 ts_tablet_manager.cc:1403] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.243s user 0.013s sys 0.021s
I20260321 02:58:17.999476 11375 tablet_service.cc:1511] Processing CreateTablet for tablet 62b43ed6f6404b7aa8463e168f98232a (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260321 02:58:18.016571 11428 raft_consensus.cc:2749] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:18.085263 11429 ts_tablet_manager.cc:1434] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.216s user 0.012s sys 0.005s
I20260321 02:58:18.094558 11442 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:18.094709 11385 raft_consensus.cc:2468] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:18.095413 11429 tablet_bootstrap.cc:492] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
W20260321 02:58:18.075320 11119 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 56612.4us
I20260321 02:58:18.095283 11442 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.098754 11200 leader_election.cc:304] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:18.099632 11375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62b43ed6f6404b7aa8463e168f98232a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.100258 11430 raft_consensus.cc:2804] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:18.100860 11299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd5fbcf5c6ce4a819d81ed0d22bf7636. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.112691 11424 raft_consensus.cc:359] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.113423 11296 tablet_service.cc:1511] Processing CreateTablet for tablet 42855a31af244db9b41f42c30304de34 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260321 02:58:18.116868 11425 tablet_bootstrap.cc:492] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:18.125010 11425 ts_tablet_manager.cc:1403] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.285s user 0.012s sys 0.012s
I20260321 02:58:18.127085 11218 tablet_service.cc:1511] Processing CreateTablet for tablet 446eeb6a12c64deb848b6db18a6587f6 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260321 02:58:18.128129 11425 raft_consensus.cc:359] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.131536 11425 raft_consensus.cc:385] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:18.129943 11374 tablet_service.cc:1511] Processing CreateTablet for tablet 446eeb6a12c64deb848b6db18a6587f6 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260321 02:58:18.133064 11374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 446eeb6a12c64deb848b6db18a6587f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.133690 11223 tablet_service.cc:1511] Processing CreateTablet for tablet 2b701d6135bc458c9aef1d5303344525 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260321 02:58:18.128510 11217 tablet_service.cc:1511] Processing CreateTablet for tablet 2d71c397418a46d085c9974230b7f964 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260321 02:58:18.131309 11225 tablet_service.cc:1511] Processing CreateTablet for tablet 00eeadd5c566459b86e97f920a91c398 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260321 02:58:18.136422 11425 raft_consensus.cc:740] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:18.137583 11425 consensus_queue.cc:260] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.137238 11295 tablet_service.cc:1511] Processing CreateTablet for tablet 62b43ed6f6404b7aa8463e168f98232a (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260321 02:58:18.143507 11429 tablet_bootstrap.cc:654] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:18.131327 11218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 446eeb6a12c64deb848b6db18a6587f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.156292 11217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d71c397418a46d085c9974230b7f964. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.157138 11372 tablet_service.cc:1511] Processing CreateTablet for tablet c2f6dd25ab2545e59d5922d37e1ec68d (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260321 02:58:18.158766 11372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2f6dd25ab2545e59d5922d37e1ec68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.160735 11225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00eeadd5c566459b86e97f920a91c398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.162148 11297 tablet_service.cc:1511] Processing CreateTablet for tablet 3b4e9b11d3c54f328ebeeee4854be727 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260321 02:58:18.160706 11296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42855a31af244db9b41f42c30304de34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.164559 11223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b701d6135bc458c9aef1d5303344525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.164939 11297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b4e9b11d3c54f328ebeeee4854be727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.166021 11442 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:18.183012 11295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62b43ed6f6404b7aa8463e168f98232a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.122393 11424 raft_consensus.cc:385] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:18.193583 11424 raft_consensus.cc:740] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:18.194666 11424 consensus_queue.cc:260] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.196200 11430 raft_consensus.cc:697] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:18.201458 11222 tablet_service.cc:1511] Processing CreateTablet for tablet 7c6a0a98407b42d18543015ce289b654 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260321 02:58:18.209534 11222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c6a0a98407b42d18543015ce289b654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.214259 11430 consensus_queue.cc:237] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.221956 11297 tablet_service.cc:1511] Processing CreateTablet for tablet 8a38bdd851a24a9a9c5e835ac457cda2 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260321 02:58:18.222287 11298 tablet_service.cc:1511] Processing CreateTablet for tablet 006e9dd81c0c4368bed549c97128f4ea (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260321 02:58:18.223445 11297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a38bdd851a24a9a9c5e835ac457cda2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.231331 11424 ts_tablet_manager.cc:1434] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.158s user 0.007s sys 0.000s
I20260321 02:58:18.232445 11424 tablet_bootstrap.cc:492] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:18.232554 11300 tablet_service.cc:1511] Processing CreateTablet for tablet c2f6dd25ab2545e59d5922d37e1ec68d (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260321 02:58:18.187510 11226 tablet_service.cc:1511] Processing CreateTablet for tablet f6a7450895514f1dad480d2716c732d9 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260321 02:58:18.192833 11368 tablet_service.cc:1511] Processing CreateTablet for tablet 2d71c397418a46d085c9974230b7f964 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260321 02:58:18.250010 11298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 006e9dd81c0c4368bed549c97128f4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.250715 11293 tablet_service.cc:1511] Processing CreateTablet for tablet 2d71c397418a46d085c9974230b7f964 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260321 02:58:18.251739 11226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6a7450895514f1dad480d2716c732d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.267333 11294 tablet_service.cc:1511] Processing CreateTablet for tablet 446eeb6a12c64deb848b6db18a6587f6 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260321 02:58:18.268028 11293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d71c397418a46d085c9974230b7f964. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.268899 11294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 446eeb6a12c64deb848b6db18a6587f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.269932 11366 tablet_service.cc:1511] Processing CreateTablet for tablet 2b701d6135bc458c9aef1d5303344525 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260321 02:58:18.258288 11367 tablet_service.cc:1511] Processing CreateTablet for tablet f6a7450895514f1dad480d2716c732d9 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260321 02:58:18.259020 11425 ts_tablet_manager.cc:1434] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.134s user 0.013s sys 0.003s
I20260321 02:58:18.274190 11300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2f6dd25ab2545e59d5922d37e1ec68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.296550 11292 tablet_service.cc:1511] Processing CreateTablet for tablet 00eeadd5c566459b86e97f920a91c398 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260321 02:58:18.297974 11292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00eeadd5c566459b86e97f920a91c398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.302119 11368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d71c397418a46d085c9974230b7f964. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.315440 11428 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:18.315228 11225 tablet_service.cc:1511] Processing CreateTablet for tablet e626ec524e8c40ec8062293d8268e85f (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260321 02:58:18.315932 11428 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.316782 11225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e626ec524e8c40ec8062293d8268e85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.317207 11299 tablet_service.cc:1511] Processing CreateTablet for tablet 2b701d6135bc458c9aef1d5303344525 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260321 02:58:18.317987 11428 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:18.318511 11299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b701d6135bc458c9aef1d5303344525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.321843 11373 tablet_service.cc:1511] Processing CreateTablet for tablet 7c6a0a98407b42d18543015ce289b654 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260321 02:58:18.324229 11366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b701d6135bc458c9aef1d5303344525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.325888 11218 tablet_service.cc:1511] Processing CreateTablet for tablet a42977f759be4144a5862197a77e395d (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260321 02:58:18.326676 11224 tablet_service.cc:1511] Processing CreateTablet for tablet 8dad9ff92f18442fa92c79c7c804610b (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260321 02:58:18.327201 11218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a42977f759be4144a5862197a77e395d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.327896 11224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dad9ff92f18442fa92c79c7c804610b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.328383 11373 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c6a0a98407b42d18543015ce289b654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.329234 11219 tablet_service.cc:1511] Processing CreateTablet for tablet 1b8d0e26d25e4888bc95c0956520bdd1 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260321 02:58:18.330781 11219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b8d0e26d25e4888bc95c0956520bdd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.291312 11371 tablet_service.cc:1511] Processing CreateTablet for tablet 00eeadd5c566459b86e97f920a91c398 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260321 02:58:18.333078 11371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00eeadd5c566459b86e97f920a91c398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.333362 11374 tablet_service.cc:1511] Processing CreateTablet for tablet e626ec524e8c40ec8062293d8268e85f (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260321 02:58:18.334757 11374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e626ec524e8c40ec8062293d8268e85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.334300 11217 tablet_service.cc:1511] Processing CreateTablet for tablet fb9aee1ed8d94adbbef4eb7ac83735f1 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260321 02:58:18.336037 11425 tablet_bootstrap.cc:492] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:18.340143 11134 catalog_manager.cc:5671] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:18.345155 11217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb9aee1ed8d94adbbef4eb7ac83735f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.352880 11370 tablet_service.cc:1511] Processing CreateTablet for tablet fb9aee1ed8d94adbbef4eb7ac83735f1 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260321 02:58:18.354208 11370 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb9aee1ed8d94adbbef4eb7ac83735f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.354952 11429 tablet_bootstrap.cc:492] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:18.355659 11429 ts_tablet_manager.cc:1403] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.261s user 0.021s sys 0.010s
I20260321 02:58:18.343601 11430 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:18.358709 11430 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.361176 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:18.364006 11220 tablet_service.cc:1511] Processing CreateTablet for tablet ce2a3b4534454095902245efe52aeb14 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260321 02:58:18.364697 11236 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:18.362084 11375 tablet_service.cc:1511] Processing CreateTablet for tablet 1b8d0e26d25e4888bc95c0956520bdd1 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260321 02:58:18.365985 11375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b8d0e26d25e4888bc95c0956520bdd1. 1 dirs total, 0 dirs full, 0 dirs failed
W20260321 02:58:18.367596 11182 auto_rebalancer.cc:247] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260321 02:58:18.368419 11424 tablet_bootstrap.cc:654] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:18.368510 11349 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:18.369405 11428 raft_consensus.cc:2804] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:18.369796 11428 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:18.370148 11428 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:18.371912 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:58:18.373389 11200 leader_election.cc:343] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:18.379159 11428 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.380399 11220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce2a3b4534454095902245efe52aeb14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.381887 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:18.392517 11385 raft_consensus.cc:2393] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 1: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:58:18.395141 11200 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:18.396329 11446 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:18.398996 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:18.402347 11428 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:18.405805 11425 tablet_bootstrap.cc:654] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:18.384783 11430 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:18.409214 11367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6a7450895514f1dad480d2716c732d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.417311 11429 raft_consensus.cc:359] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.418205 11429 raft_consensus.cc:385] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:18.418558 11429 raft_consensus.cc:740] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:18.419593 11429 consensus_queue.cc:260] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.422781 11429 ts_tablet_manager.cc:1434] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.064s user 0.006s sys 0.000s
I20260321 02:58:18.428781 11375 tablet_service.cc:1511] Processing CreateTablet for tablet a42977f759be4144a5862197a77e395d (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260321 02:58:18.430339 11375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a42977f759be4144a5862197a77e395d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.437111 11374 tablet_service.cc:1511] Processing CreateTablet for tablet 8dad9ff92f18442fa92c79c7c804610b (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260321 02:58:18.439085 11374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dad9ff92f18442fa92c79c7c804610b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.439355 11429 tablet_bootstrap.cc:492] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:18.448295 11222 tablet_service.cc:1511] Processing CreateTablet for tablet 96159c887ede40b68c4dc0ab508ada79 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260321 02:58:18.449656 11222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96159c887ede40b68c4dc0ab508ada79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.454440 11372 tablet_service.cc:1511] Processing CreateTablet for tablet ce2a3b4534454095902245efe52aeb14 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260321 02:58:18.438879 11446 raft_consensus.cc:2749] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:18.460979 11372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce2a3b4534454095902245efe52aeb14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.489032 11429 tablet_bootstrap.cc:654] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:18.510520 11366 tablet_service.cc:1511] Processing CreateTablet for tablet 96159c887ede40b68c4dc0ab508ada79 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260321 02:58:18.512178 11366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96159c887ede40b68c4dc0ab508ada79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.530637 11424 tablet_bootstrap.cc:492] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:18.532819 11424 ts_tablet_manager.cc:1403] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.301s user 0.017s sys 0.017s
I20260321 02:58:18.545272 11294 tablet_service.cc:1511] Processing CreateTablet for tablet e626ec524e8c40ec8062293d8268e85f (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260321 02:58:18.547230 11296 tablet_service.cc:1511] Processing CreateTablet for tablet fb9aee1ed8d94adbbef4eb7ac83735f1 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260321 02:58:18.548174 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:18.549057 11301 tablet_service.cc:1511] Processing CreateTablet for tablet 1b8d0e26d25e4888bc95c0956520bdd1 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260321 02:58:18.543684 11225 tablet_service.cc:1511] Processing CreateTablet for tablet 86406dbef163402197aa42a22bcc0101 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260321 02:58:18.551007 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:18.551882 11225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86406dbef163402197aa42a22bcc0101. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.552102 11442 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:18.552639 11442 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.554786 11442 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:18.556341 11297 tablet_service.cc:1511] Processing CreateTablet for tablet 7c6a0a98407b42d18543015ce289b654 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260321 02:58:18.558568 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:18.561744 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:18.563614 11294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e626ec524e8c40ec8062293d8268e85f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.559898 11298 tablet_service.cc:1511] Processing CreateTablet for tablet ce2a3b4534454095902245efe52aeb14 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260321 02:58:18.571635 11301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b8d0e26d25e4888bc95c0956520bdd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.569628 11424 raft_consensus.cc:359] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.576897 11424 raft_consensus.cc:385] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:18.577251 11424 raft_consensus.cc:740] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:18.577389 11295 tablet_service.cc:1511] Processing CreateTablet for tablet 96159c887ede40b68c4dc0ab508ada79 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260321 02:58:18.580484 11429 tablet_bootstrap.cc:492] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:18.581128 11429 ts_tablet_manager.cc:1403] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.158s user 0.013s sys 0.006s
I20260321 02:58:18.581624 11425 tablet_bootstrap.cc:492] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:18.582134 11425 ts_tablet_manager.cc:1403] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.246s user 0.034s sys 0.037s
I20260321 02:58:18.585168 11425 raft_consensus.cc:359] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.585523 11368 tablet_service.cc:1511] Processing CreateTablet for tablet 86406dbef163402197aa42a22bcc0101 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260321 02:58:18.586046 11425 raft_consensus.cc:385] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:18.586386 11425 raft_consensus.cc:740] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:18.587199 11368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86406dbef163402197aa42a22bcc0101. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.587363 11425 consensus_queue.cc:260] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.591009 11425 ts_tablet_manager.cc:1434] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20260321 02:58:18.591800 11429 raft_consensus.cc:359] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.594267 11371 tablet_service.cc:1511] Processing CreateTablet for tablet b270af8b9da943d2b91acbdc0bd603cc (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260321 02:58:18.597007 11236 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:18.598114 11372 tablet_service.cc:1511] Processing CreateTablet for tablet 3c50eb057ef548839834ecb7dfd20ce5 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260321 02:58:18.598830 11226 tablet_service.cc:1511] Processing CreateTablet for tablet 2f5e054aa3c74a3282b3340fd3b1ea21 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260321 02:58:18.597056 11371 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b270af8b9da943d2b91acbdc0bd603cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.599437 11349 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:18.600287 11226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f5e054aa3c74a3282b3340fd3b1ea21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.600378 11372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c50eb057ef548839834ecb7dfd20ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.600489 11428 raft_consensus.cc:2804] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:18.595072 11219 tablet_service.cc:1511] Processing CreateTablet for tablet 3c50eb057ef548839834ecb7dfd20ce5 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260321 02:58:18.603293 11219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c50eb057ef548839834ecb7dfd20ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.597010 11425 tablet_bootstrap.cc:492] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:18.605332 11429 raft_consensus.cc:385] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:18.607658 11374 tablet_service.cc:1511] Processing CreateTablet for tablet 2f5e054aa3c74a3282b3340fd3b1ea21 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260321 02:58:18.610016 11374 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f5e054aa3c74a3282b3340fd3b1ea21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.612730 11297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c6a0a98407b42d18543015ce289b654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.614797 11296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb9aee1ed8d94adbbef4eb7ac83735f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.616052 11429 raft_consensus.cc:740] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:18.629755 11300 tablet_service.cc:1511] Processing CreateTablet for tablet a42977f759be4144a5862197a77e395d (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260321 02:58:18.632229 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:18.632431 11292 tablet_service.cc:1511] Processing CreateTablet for tablet 8dad9ff92f18442fa92c79c7c804610b (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
W20260321 02:58:18.633222 11275 leader_election.cc:343] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
W20260321 02:58:18.633400 11349 leader_election.cc:343] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:18.633539 11429 consensus_queue.cc:260] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:18.633997 11275 leader_election.cc:343] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
W20260321 02:58:18.634330 11349 leader_election.cc:343] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:18.634429 11275 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:18.617142 11293 tablet_service.cc:1511] Processing CreateTablet for tablet f6a7450895514f1dad480d2716c732d9 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260321 02:58:18.635876 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:18.636368 11444 raft_consensus.cc:2749] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:18.636593 11295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96159c887ede40b68c4dc0ab508ada79. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.635911 11429 ts_tablet_manager.cc:1434] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.054s user 0.007s sys 0.000s
I20260321 02:58:18.648386 11428 raft_consensus.cc:697] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:18.578305 11424 consensus_queue.cc:260] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.652112 11425 tablet_bootstrap.cc:654] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:18.651412 11428 consensus_queue.cc:237] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.659195 11299 tablet_service.cc:1511] Processing CreateTablet for tablet 2f5e054aa3c74a3282b3340fd3b1ea21 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260321 02:58:18.656747 11292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8dad9ff92f18442fa92c79c7c804610b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.661192 11372 tablet_service.cc:1511] Processing CreateTablet for tablet ff63bfd190b741c98793ff65c77265da (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260321 02:58:18.661972 11299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f5e054aa3c74a3282b3340fd3b1ea21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.662818 11372 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff63bfd190b741c98793ff65c77265da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.664232 11429 tablet_bootstrap.cc:492] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:18.665005 11298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce2a3b4534454095902245efe52aeb14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.668272 11300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a42977f759be4144a5862197a77e395d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.668490 11220 tablet_service.cc:1511] Processing CreateTablet for tablet b270af8b9da943d2b91acbdc0bd603cc (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260321 02:58:18.669862 11220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b270af8b9da943d2b91acbdc0bd603cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.683056 11217 tablet_service.cc:1511] Processing CreateTablet for tablet ff63bfd190b741c98793ff65c77265da (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260321 02:58:18.689989 11217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff63bfd190b741c98793ff65c77265da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.692104 11293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6a7450895514f1dad480d2716c732d9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260321 02:58:18.656240 11275 leader_election.cc:343] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
W20260321 02:58:18.701828 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 47022.3us
W20260321 02:58:18.703298 11275 leader_election.cc:343] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:18.709156 11424 ts_tablet_manager.cc:1434] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.176s user 0.003s sys 0.007s
I20260321 02:58:18.710279 11424 tablet_bootstrap.cc:492] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:18.710232 11294 tablet_service.cc:1511] Processing CreateTablet for tablet b270af8b9da943d2b91acbdc0bd603cc (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260321 02:58:18.711723 11294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b270af8b9da943d2b91acbdc0bd603cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.733119 11442 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:18.733690 11442 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.736924 11442 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:18.703902 11275 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:18.738101 11442 raft_consensus.cc:2749] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:18.739279 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:58:18.742514 11275 leader_election.cc:343] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:18.744361 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:18.745240 11385 raft_consensus.cc:2468] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
I20260321 02:58:18.746709 11275 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:18.748492 11429 tablet_bootstrap.cc:654] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:18.748448 11442 raft_consensus.cc:2804] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:18.748960 11442 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:18.749326 11442 raft_consensus.cc:3060] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:18.759395 11442 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.767168 11301 tablet_service.cc:1511] Processing CreateTablet for tablet 86406dbef163402197aa42a22bcc0101 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260321 02:58:18.767753 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:18.768967 11301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86406dbef163402197aa42a22bcc0101. 1 dirs total, 0 dirs full, 0 dirs failed
W20260321 02:58:18.769440 11275 leader_election.cc:343] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:18.771484 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:18.772101 11385 raft_consensus.cc:3060] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:18.762714 11296 tablet_service.cc:1511] Processing CreateTablet for tablet 3c50eb057ef548839834ecb7dfd20ce5 (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
W20260321 02:58:18.777534 11200 consensus_peers.cc:597] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb -> Peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Couldn't send request to peer 83b12b883a7f4129a87dadf5a78f08da. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:18.779757 11296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c50eb057ef548839834ecb7dfd20ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.777750 11299 tablet_service.cc:1511] Processing CreateTablet for tablet ff63bfd190b741c98793ff65c77265da (DEFAULT_TABLE table=test-workload [id=07ae567e73b94faa9ae558018daa9713]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260321 02:58:18.780855 11442 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:18.780428 11385 raft_consensus.cc:2468] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:18.785182 11275 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:18.786713 11299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff63bfd190b741c98793ff65c77265da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:18.787796 11442 raft_consensus.cc:2804] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:18.794304 11442 raft_consensus.cc:697] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:58:18.795310 11442 consensus_queue.cc:237] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.798204 11424 tablet_bootstrap.cc:654] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:18.804565 11429 tablet_bootstrap.cc:492] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:18.805099 11429 ts_tablet_manager.cc:1403] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.141s user 0.018s sys 0.001s
I20260321 02:58:18.807562 11429 raft_consensus.cc:359] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.811129 11429 raft_consensus.cc:385] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:18.813920 11425 tablet_bootstrap.cc:492] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:18.814414 11425 ts_tablet_manager.cc:1403] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.218s user 0.018s sys 0.010s
I20260321 02:58:18.817828 11425 raft_consensus.cc:359] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.818568 11425 raft_consensus.cc:385] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:18.818868 11425 raft_consensus.cc:740] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:18.811661 11429 raft_consensus.cc:740] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:18.824286 11429 consensus_queue.cc:260] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.825006 11134 catalog_manager.cc:5671] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:18.819617 11425 consensus_queue.cc:260] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.829489 11425 ts_tablet_manager.cc:1434] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.015s user 0.008s sys 0.001s
I20260321 02:58:18.830539 11425 tablet_bootstrap.cc:492] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:18.831269 11429 ts_tablet_manager.cc:1434] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.026s user 0.008s sys 0.000s
I20260321 02:58:18.832305 11429 tablet_bootstrap.cc:492] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:18.835093 11446 consensus_queue.cc:1048] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:18.838475 11425 tablet_bootstrap.cc:654] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:18.848456 11429 tablet_bootstrap.cc:654] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:18.871138 11425 tablet_bootstrap.cc:492] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:18.871832 11425 ts_tablet_manager.cc:1403] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.041s user 0.018s sys 0.010s
I20260321 02:58:18.880589 11442 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:18.886060 11442 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.888733 11442 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:18.889650 11134 catalog_manager.cc:5671] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:18.890563 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:18.893080 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:58:18.894673 11275 leader_election.cc:343] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
W20260321 02:58:18.896513 11275 leader_election.cc:343] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:18.896957 11275 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:18.904475 11442 raft_consensus.cc:2749] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:18.903779 11424 tablet_bootstrap.cc:492] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:18.899443 11425 raft_consensus.cc:359] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.906344 11424 ts_tablet_manager.cc:1403] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.196s user 0.028s sys 0.022s
I20260321 02:58:18.915264 11424 raft_consensus.cc:359] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.916098 11424 raft_consensus.cc:385] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:18.916424 11424 raft_consensus.cc:740] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:18.907513 11425 raft_consensus.cc:385] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:18.917771 11425 raft_consensus.cc:740] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:18.918534 11425 consensus_queue.cc:260] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.920866 11425 ts_tablet_manager.cc:1434] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.044s user 0.007s sys 0.001s
I20260321 02:58:18.921746 11425 tablet_bootstrap.cc:492] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:18.911876 11429 tablet_bootstrap.cc:492] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:18.928337 11429 ts_tablet_manager.cc:1403] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.096s user 0.008s sys 0.014s
I20260321 02:58:18.931169 11428 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:18.931783 11429 raft_consensus.cc:359] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.931870 11428 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.933090 11424 consensus_queue.cc:260] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.935328 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:18.936075 11236 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:18.936995 11424 ts_tablet_manager.cc:1434] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.025s user 0.006s sys 0.001s
I20260321 02:58:18.937350 11428 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:18.937193 11349 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:18.937794 11425 tablet_bootstrap.cc:654] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:18.938107 11428 raft_consensus.cc:2804] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:18.938917 11428 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:18.939090 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:18.939656 11424 tablet_bootstrap.cc:492] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:18.939291 11428 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
W20260321 02:58:18.940474 11349 leader_election.cc:343] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:18.945111 11428 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.946102 11424 tablet_bootstrap.cc:654] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:18.932605 11429 raft_consensus.cc:385] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:18.946535 11429 raft_consensus.cc:740] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:18.949888 11429 consensus_queue.cc:260] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.952860 11429 ts_tablet_manager.cc:1434] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.024s user 0.007s sys 0.001s
I20260321 02:58:18.948483 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:18.954581 11429 tablet_bootstrap.cc:492] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:18.954620 11236 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:18.957247 11425 tablet_bootstrap.cc:492] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:18.957779 11425 ts_tablet_manager.cc:1403] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.007s
I20260321 02:58:18.958993 11424 tablet_bootstrap.cc:492] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:18.959439 11424 ts_tablet_manager.cc:1403] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.000s
I20260321 02:58:18.962603 11429 tablet_bootstrap.cc:654] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:18.962524 11424 raft_consensus.cc:359] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.963721 11424 raft_consensus.cc:385] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:18.964061 11424 raft_consensus.cc:740] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:18.964982 11424 consensus_queue.cc:260] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.967054 11236 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:18.968326 11424 ts_tablet_manager.cc:1434] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.008s user 0.010s sys 0.000s
I20260321 02:58:18.968235 11425 raft_consensus.cc:359] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.968995 11425 raft_consensus.cc:385] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:18.969307 11425 raft_consensus.cc:740] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:18.969437 11424 tablet_bootstrap.cc:492] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:18.970108 11425 consensus_queue.cc:260] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.976720 11424 tablet_bootstrap.cc:654] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:18.982193 11425 ts_tablet_manager.cc:1434] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.024s user 0.007s sys 0.000s
I20260321 02:58:18.985565 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:58:18.987532 11349 leader_election.cc:343] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:18.988385 11428 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:18.989199 11424 tablet_bootstrap.cc:492] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:18.989696 11424 ts_tablet_manager.cc:1403] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.021s user 0.013s sys 0.005s
I20260321 02:58:18.989940 11349 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:18.991545 11428 raft_consensus.cc:2804] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:18.992269 11425 tablet_bootstrap.cc:492] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:18.992669 11424 raft_consensus.cc:359] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.993831 11424 raft_consensus.cc:385] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:18.993321 11428 raft_consensus.cc:697] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:18.994151 11424 raft_consensus.cc:740] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:18.995085 11424 consensus_queue.cc:260] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.995335 11428 consensus_queue.cc:237] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:18.997900 11424 ts_tablet_manager.cc:1434] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20260321 02:58:18.998943 11424 tablet_bootstrap.cc:492] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:19.000108 11425 tablet_bootstrap.cc:654] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
W20260321 02:58:19.004834 11262 tablet.cc:2379] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260321 02:58:19.009224 11425 tablet_bootstrap.cc:492] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:19.009718 11425 ts_tablet_manager.cc:1403] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.004s
I20260321 02:58:19.011168 11429 tablet_bootstrap.cc:492] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:19.011619 11424 tablet_bootstrap.cc:654] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.011693 11429 ts_tablet_manager.cc:1403] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.058s user 0.031s sys 0.015s
I20260321 02:58:19.012300 11425 raft_consensus.cc:359] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.011677 11132 catalog_manager.cc:5671] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:19.013041 11425 raft_consensus.cc:385] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.013345 11425 raft_consensus.cc:740] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.014048 11425 consensus_queue.cc:260] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.016582 11429 raft_consensus.cc:359] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.017546 11429 raft_consensus.cc:385] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.017969 11429 raft_consensus.cc:740] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.019435 11429 consensus_queue.cc:260] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.026361 11429 ts_tablet_manager.cc:1434] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20260321 02:58:19.027717 11429 tablet_bootstrap.cc:492] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:19.027878 11425 ts_tablet_manager.cc:1434] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260321 02:58:19.038144 11429 tablet_bootstrap.cc:654] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.041445 11425 tablet_bootstrap.cc:492] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:19.045464 11446 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:19.045984 11446 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.050026 11446 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:19.050876 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:19.051590 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:19.052703 11385 raft_consensus.cc:2468] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
W20260321 02:58:19.052802 11200 leader_election.cc:343] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:19.053032 11425 tablet_bootstrap.cc:654] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.054664 11200 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:19.055361 11424 tablet_bootstrap.cc:492] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:19.055806 11424 ts_tablet_manager.cc:1403] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.057s user 0.023s sys 0.020s
I20260321 02:58:19.055747 11446 raft_consensus.cc:2804] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:19.056210 11446 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:19.056598 11446 raft_consensus.cc:3060] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.056951 11429 tablet_bootstrap.cc:492] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:19.057634 11429 ts_tablet_manager.cc:1403] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.009s
I20260321 02:58:19.058375 11424 raft_consensus.cc:359] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.059029 11424 raft_consensus.cc:385] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.059286 11424 raft_consensus.cc:740] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.059932 11424 consensus_queue.cc:260] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.061626 11429 raft_consensus.cc:359] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.062652 11429 raft_consensus.cc:385] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.062963 11430 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:19.063027 11429 raft_consensus.cc:740] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.063510 11430 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.064262 11429 consensus_queue.cc:260] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.065474 11425 tablet_bootstrap.cc:492] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:19.065428 11430 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:19.066022 11425 ts_tablet_manager.cc:1403] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.002s
I20260321 02:58:19.066979 11429 ts_tablet_manager.cc:1434] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260321 02:58:19.067886 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:19.068540 11309 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:19.068835 11429 tablet_bootstrap.cc:492] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:19.069509 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:19.069667 11446 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.071321 11200 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:19.071633 11446 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:19.062664 11424 ts_tablet_manager.cc:1434] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260321 02:58:19.072569 11428 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:19.073091 11428 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.073616 11424 tablet_bootstrap.cc:492] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:19.074393 11309 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.074930 11428 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:58:19.076547 11200 leader_election.cc:343] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:19.077195 11429 tablet_bootstrap.cc:654] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.077310 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:19.078501 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:58:19.079064 11349 leader_election.cc:343] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:19.080621 11424 tablet_bootstrap.cc:654] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.082410 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:19.083082 11385 raft_consensus.cc:3060] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.083510 11446 raft_consensus.cc:2804] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:19.083874 11446 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:19.084213 11446 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
W20260321 02:58:19.089133 11200 leader_election.cc:343] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:19.091836 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:19.097206 11425 raft_consensus.cc:359] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.097913 11425 raft_consensus.cc:385] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.098038 11424 tablet_bootstrap.cc:492] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:19.098234 11425 raft_consensus.cc:740] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.095700 11429 tablet_bootstrap.cc:492] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:19.099289 11429 ts_tablet_manager.cc:1403] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.012s
I20260321 02:58:19.091928 11446 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:19.099992 11349 leader_election.cc:343] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:19.100450 11349 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:19.101455 11428 raft_consensus.cc:2749] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:19.101851 11429 raft_consensus.cc:359] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.102572 11429 raft_consensus.cc:385] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.102602 11446 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:19.098595 11424 ts_tablet_manager.cc:1403] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.001s
I20260321 02:58:19.103120 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:19.103727 11311 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.104075 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:58:19.105310 11200 leader_election.cc:343] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:19.105818 11424 raft_consensus.cc:359] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.106424 11424 raft_consensus.cc:385] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.106593 11424 raft_consensus.cc:740] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.107123 11424 consensus_queue.cc:260] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.102911 11429 raft_consensus.cc:740] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.109054 11424 ts_tablet_manager.cc:1434] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260321 02:58:19.109308 11429 consensus_queue.cc:260] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.109768 11425 consensus_queue.cc:260] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.110697 11428 consensus_queue.cc:1048] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:19.112138 11425 ts_tablet_manager.cc:1434] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.022s user 0.007s sys 0.001s
I20260321 02:58:19.113500 11425 tablet_bootstrap.cc:492] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:19.117053 11424 tablet_bootstrap.cc:492] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:19.120174 11425 tablet_bootstrap.cc:654] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.121083 11442 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:19.121567 11442 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.123696 11442 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:19.126062 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:19.126765 11236 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
I20260321 02:58:19.128157 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:19.128983 11384 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
I20260321 02:58:19.130319 11275 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:19.130355 11446 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:19.131599 11446 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.133922 11446 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:19.135406 11442 raft_consensus.cc:2804] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:19.135807 11442 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:19.136171 11442 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.136693 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:19.111888 11311 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
W20260321 02:58:19.138768 11200 leader_election.cc:343] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:19.139810 11200 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:19.141463 11429 ts_tablet_manager.cc:1434] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.042s user 0.005s sys 0.001s
I20260321 02:58:19.141635 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:19.120100 11385 raft_consensus.cc:2468] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:19.143569 11446 raft_consensus.cc:2804] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:19.143823 11429 tablet_bootstrap.cc:492] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:19.142554 11384 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:19.144181 11442 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.145154 11446 raft_consensus.cc:697] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:19.146214 11446 consensus_queue.cc:237] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.147264 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:19.147897 11236 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.150844 11429 tablet_bootstrap.cc:654] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.150465 11200 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:19.151695 11430 raft_consensus.cc:2804] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:19.152823 11428 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:19.153544 11428 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.157249 11424 tablet_bootstrap.cc:654] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.165437 11236 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:19.167006 11275 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:19.168179 11444 raft_consensus.cc:2804] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:19.157269 11428 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:19.171559 11200 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:19.174880 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:19.170740 11442 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:19.176164 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:19.180089 11311 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:19.186160 11444 raft_consensus.cc:697] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:58:19.187534 11487 raft_consensus.cc:2804] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:19.182600 11384 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.187427 11444 consensus_queue.cc:237] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.187875 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:19.191895 11425 tablet_bootstrap.cc:492] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:19.192515 11425 ts_tablet_manager.cc:1403] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.080s user 0.017s sys 0.022s
I20260321 02:58:19.193441 11349 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:19.194702 11428 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260321 02:58:19.194970 11349 leader_election.cc:343] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:19.195160 11428 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:19.195535 11428 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.197218 11458 consensus_queue.cc:1048] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:19.183916 11430 raft_consensus.cc:697] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:19.201866 11425 raft_consensus.cc:359] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.206086 11425 raft_consensus.cc:385] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.206379 11425 raft_consensus.cc:740] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.196874 11486 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:19.206971 11425 consensus_queue.cc:260] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.209553 11430 consensus_queue.cc:237] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.210199 11425 ts_tablet_manager.cc:1434] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260321 02:58:19.198662 11487 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:19.210901 11487 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.211056 11425 tablet_bootstrap.cc:492] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:19.221958 11446 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:19.235167 11424 tablet_bootstrap.cc:492] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:19.235651 11424 ts_tablet_manager.cc:1403] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.119s user 0.013s sys 0.016s
I20260321 02:58:19.217278 11486 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.224046 11446 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.240236 11424 raft_consensus.cc:359] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } }
I20260321 02:58:19.240778 11424 raft_consensus.cc:385] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.240999 11424 raft_consensus.cc:740] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.241498 11424 consensus_queue.cc:260] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } }
I20260321 02:58:19.242859 11484 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:19.243767 11484 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.232477 11130 catalog_manager.cc:5671] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:19.253391 11384 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:19.258657 11446 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:19.259508 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:19.260116 11429 tablet_bootstrap.cc:492] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:19.261756 11424 ts_tablet_manager.cc:1434] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.026s user 0.005s sys 0.001s
I20260321 02:58:19.262624 11424 tablet_bootstrap.cc:492] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:19.260573 11429 ts_tablet_manager.cc:1403] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.117s user 0.009s sys 0.013s
I20260321 02:58:19.268675 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:19.269948 11425 tablet_bootstrap.cc:654] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.267300 11429 raft_consensus.cc:359] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } }
I20260321 02:58:19.273569 11429 raft_consensus.cc:385] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.273862 11429 raft_consensus.cc:740] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.274586 11429 consensus_queue.cc:260] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } }
I20260321 02:58:19.278009 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:19.256528 11428 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.282979 11429 ts_tablet_manager.cc:1434] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.020s user 0.004s sys 0.002s
I20260321 02:58:19.283943 11429 tablet_bootstrap.cc:492] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:19.284847 11311 raft_consensus.cc:3060] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.288333 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:19.289105 11309 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:19.290073 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:19.290866 11385 raft_consensus.cc:2393] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 1: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:19.290284 11200 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:19.292364 11275 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:19.293236 11444 raft_consensus.cc:3060] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.294399 11430 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:19.280963 11131 catalog_manager.cc:5671] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:19.294977 11430 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:19.295058 11484 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:19.295419 11430 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.282255 11486 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:19.301455 11428 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:19.303651 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:19.299373 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:19.305064 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:19.306684 11309 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.306975 11383 raft_consensus.cc:2393] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 1: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:58:19.311206 11425 tablet_bootstrap.cc:492] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:19.311568 11425 ts_tablet_manager.cc:1403] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.101s user 0.006s sys 0.007s
I20260321 02:58:19.302696 11384 raft_consensus.cc:2393] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 1: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:19.311556 11424 tablet_bootstrap.cc:654] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.310722 11133 catalog_manager.cc:5671] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:19.317147 11429 tablet_bootstrap.cc:654] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.317924 11200 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:19.320080 11484 raft_consensus.cc:3060] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.319607 11425 raft_consensus.cc:359] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.321514 11425 raft_consensus.cc:385] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.321753 11425 raft_consensus.cc:740] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.326714 11425 consensus_queue.cc:260] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.320824 11486 consensus_queue.cc:1048] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260321 02:58:19.331697 11425 ts_tablet_manager.cc:1434] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20260321 02:58:19.332533 11425 tablet_bootstrap.cc:492] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:19.329522 11444 raft_consensus.cc:2749] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:19.352483 11484 raft_consensus.cc:2749] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:19.333137 11487 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.360736 11309 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:19.362005 11430 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.364277 11487 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:19.364691 11236 raft_consensus.cc:2393] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 1: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:19.365402 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:19.365482 11349 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:19.366051 11384 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.367209 11428 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:19.367728 11428 raft_consensus.cc:697] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:19.369519 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:19.373035 11429 tablet_bootstrap.cc:492] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:19.371557 11486 consensus_queue.cc:1048] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:19.373473 11429 ts_tablet_manager.cc:1403] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.090s user 0.005s sys 0.017s
I20260321 02:58:19.374385 11425 tablet_bootstrap.cc:654] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.373924 11428 consensus_queue.cc:237] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:19.370154 11182 auto_rebalancer.cc:247] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260321 02:58:19.367671 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:19.370968 11430 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:19.377676 11429 raft_consensus.cc:359] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.378362 11429 raft_consensus.cc:385] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.378631 11429 raft_consensus.cc:740] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.379283 11429 consensus_queue.cc:260] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.380105 11424 tablet_bootstrap.cc:492] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:19.380594 11424 ts_tablet_manager.cc:1403] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.118s user 0.019s sys 0.018s
I20260321 02:58:19.382963 11429 ts_tablet_manager.cc:1434] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260321 02:58:19.383713 11429 tablet_bootstrap.cc:492] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
W20260321 02:58:19.385139 11200 leader_election.cc:343] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:19.386442 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:19.387624 11311 raft_consensus.cc:2393] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 1: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:58:19.394057 11200 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:19.394892 11487 raft_consensus.cc:2749] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260321 02:58:19.401312 11384 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:19.402948 11200 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:19.409783 11429 tablet_bootstrap.cc:654] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.403505 11424 raft_consensus.cc:359] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.409053 11130 catalog_manager.cc:5671] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:19.410223 11424 raft_consensus.cc:385] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.410843 11424 raft_consensus.cc:740] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.404171 11458 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:19.411761 11424 consensus_queue.cc:260] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.413913 11458 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.415582 11424 ts_tablet_manager.cc:1434] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.035s user 0.008s sys 0.000s
I20260321 02:58:19.416455 11458 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:19.416596 11424 tablet_bootstrap.cc:492] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:19.413499 11487 raft_consensus.cc:2804] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:19.419726 11487 raft_consensus.cc:697] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:19.417625 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:19.418411 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:19.421140 11487 consensus_queue.cc:237] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:19.423045 11349 leader_election.cc:343] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:19.421300 11425 tablet_bootstrap.cc:492] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:19.424634 11425 ts_tablet_manager.cc:1403] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.092s user 0.005s sys 0.008s
I20260321 02:58:19.425442 11311 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:19.428946 11349 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:19.430785 11458 raft_consensus.cc:2804] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:19.431195 11458 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:19.431640 11458 raft_consensus.cc:3060] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.431783 11424 tablet_bootstrap.cc:654] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.448112 11458 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.452016 11444 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:19.451917 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:19.452564 11444 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } }
I20260321 02:58:19.454676 11458 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:19.454700 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:19.456059 11311 raft_consensus.cc:3060] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.460485 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:19.461299 11384 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
I20260321 02:58:19.462935 11275 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:19.464067 11442 raft_consensus.cc:2804] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:19.464582 11442 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:19.465057 11442 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.428145 11425 raft_consensus.cc:359] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.472263 11425 raft_consensus.cc:385] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.472694 11425 raft_consensus.cc:740] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.472780 11444 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717)
I20260321 02:58:19.473832 11425 consensus_queue.cc:260] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.474882 11236 raft_consensus.cc:3060] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.474911 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:19.475627 11235 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
I20260321 02:58:19.479504 11430 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:19.480218 11430 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.482997 11430 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:19.484792 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:19.485392 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:58:19.487164 11200 leader_election.cc:343] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
W20260321 02:58:19.488107 11200 leader_election.cc:343] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:19.488663 11200 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:19.489322 11236 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:19.490558 11430 raft_consensus.cc:2749] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:19.494603 11425 ts_tablet_manager.cc:1434] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.070s user 0.007s sys 0.000s
I20260321 02:58:19.494796 11311 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:19.496389 11349 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:19.497299 11458 raft_consensus.cc:2804] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:19.497778 11458 raft_consensus.cc:697] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:19.498571 11458 consensus_queue.cc:237] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.495810 11425 tablet_bootstrap.cc:492] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:19.503728 11429 tablet_bootstrap.cc:492] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:19.504287 11429 ts_tablet_manager.cc:1403] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.121s user 0.013s sys 0.018s
I20260321 02:58:19.498605 11442 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } }
I20260321 02:58:19.506845 11429 raft_consensus.cc:359] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.507579 11429 raft_consensus.cc:385] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.507827 11429 raft_consensus.cc:740] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.508941 11424 tablet_bootstrap.cc:492] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:19.508708 11429 consensus_queue.cc:260] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.509435 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:19.509717 11424 ts_tablet_manager.cc:1403] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.093s user 0.040s sys 0.014s
I20260321 02:58:19.510010 11384 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.512372 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:19.513073 11236 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.517918 11384 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:19.519093 11275 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:19.520221 11444 raft_consensus.cc:2804] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:19.520817 11444 raft_consensus.cc:697] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:58:19.521808 11444 consensus_queue.cc:237] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } }
I20260321 02:58:19.527350 11429 ts_tablet_manager.cc:1434] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.023s user 0.004s sys 0.000s
I20260321 02:58:19.529321 11429 tablet_bootstrap.cc:492] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:19.530192 11424 raft_consensus.cc:359] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.530035 11130 catalog_manager.cc:5671] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:19.531113 11424 raft_consensus.cc:385] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.531596 11424 raft_consensus.cc:740] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.532805 11425 tablet_bootstrap.cc:654] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.522192 11236 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:19.532379 11424 consensus_queue.cc:260] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.535408 11442 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717)
I20260321 02:58:19.534617 11131 catalog_manager.cc:5671] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:19.539350 11425 tablet_bootstrap.cc:492] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:19.536116 11424 ts_tablet_manager.cc:1434] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.026s user 0.010s sys 0.000s
I20260321 02:58:19.539746 11425 ts_tablet_manager.cc:1403] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.044s user 0.011s sys 0.000s
I20260321 02:58:19.540457 11424 tablet_bootstrap.cc:492] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:19.542851 11425 raft_consensus.cc:359] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.543541 11425 raft_consensus.cc:385] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.543823 11425 raft_consensus.cc:740] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.544751 11425 consensus_queue.cc:260] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.545572 11487 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:19.546061 11487 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.549183 11424 tablet_bootstrap.cc:654] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.550110 11425 ts_tablet_manager.cc:1434] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260321 02:58:19.550987 11425 tablet_bootstrap.cc:492] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:19.551936 11429 tablet_bootstrap.cc:654] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.557444 11487 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:19.559278 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:19.559576 11458 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:19.559834 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:19.560096 11458 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.562069 11458 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:19.559305 11311 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
W20260321 02:58:19.564793 11200 leader_election.cc:343] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:19.564926 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:58:19.565500 11200 leader_election.cc:343] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:19.566346 11200 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:19.567201 11487 raft_consensus.cc:2749] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:19.568542 11425 tablet_bootstrap.cc:654] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.568266 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:19.569198 11309 raft_consensus.cc:2393] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 1: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:19.570428 11349 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:19.571000 11130 catalog_manager.cc:5671] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:19.571800 11458 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.573750 11428 consensus_queue.cc:1048] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:19.578311 11424 tablet_bootstrap.cc:492] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:19.578915 11424 ts_tablet_manager.cc:1403] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.039s user 0.014s sys 0.000s
I20260321 02:58:19.581729 11424 raft_consensus.cc:359] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.582597 11424 raft_consensus.cc:385] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.583191 11424 raft_consensus.cc:740] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.583973 11424 consensus_queue.cc:260] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.587178 11424 ts_tablet_manager.cc:1434] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260321 02:58:19.589195 11458 raft_consensus.cc:2749] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:19.591436 11424 tablet_bootstrap.cc:492] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:19.605939 11458 consensus_queue.cc:1048] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:19.609125 11444 consensus_queue.cc:1048] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:19.613623 11424 tablet_bootstrap.cc:654] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.618893 11429 tablet_bootstrap.cc:492] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:19.619431 11425 tablet_bootstrap.cc:492] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:19.619525 11429 ts_tablet_manager.cc:1403] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.091s user 0.021s sys 0.014s
I20260321 02:58:19.620208 11425 ts_tablet_manager.cc:1403] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.069s user 0.022s sys 0.012s
I20260321 02:58:19.625089 11425 raft_consensus.cc:359] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.625399 11429 raft_consensus.cc:359] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.626049 11425 raft_consensus.cc:385] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.626344 11429 raft_consensus.cc:385] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.626448 11425 raft_consensus.cc:740] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.626821 11429 raft_consensus.cc:740] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.627751 11425 consensus_queue.cc:260] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.630086 11429 consensus_queue.cc:260] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.631748 11425 ts_tablet_manager.cc:1434] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260321 02:58:19.633174 11429 ts_tablet_manager.cc:1434] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.013s user 0.006s sys 0.002s
I20260321 02:58:19.633298 11425 tablet_bootstrap.cc:492] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:19.634488 11429 tablet_bootstrap.cc:492] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:19.645126 11425 tablet_bootstrap.cc:654] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.684180 11487 consensus_queue.cc:1048] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260321 02:58:19.684180 11444 consensus_queue.cc:1048] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:19.698125 11429 tablet_bootstrap.cc:654] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.706516 11425 tablet_bootstrap.cc:492] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:19.707163 11425 ts_tablet_manager.cc:1403] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.074s user 0.024s sys 0.023s
W20260321 02:58:19.707705 11513 log.cc:927] Time spent T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.051s user 0.002s sys 0.000s
I20260321 02:58:19.710988 11425 raft_consensus.cc:359] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.711717 11425 raft_consensus.cc:385] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.712128 11425 raft_consensus.cc:740] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.712848 11425 consensus_queue.cc:260] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.715068 11425 ts_tablet_manager.cc:1434] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260321 02:58:19.715981 11425 tablet_bootstrap.cc:492] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:19.716883 11424 tablet_bootstrap.cc:492] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:19.717341 11424 ts_tablet_manager.cc:1403] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.126s user 0.040s sys 0.029s
I20260321 02:58:19.721118 11424 raft_consensus.cc:359] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.722115 11424 raft_consensus.cc:385] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.722471 11424 raft_consensus.cc:740] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.723330 11424 consensus_queue.cc:260] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.727738 11424 ts_tablet_manager.cc:1434] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.010s user 0.006s sys 0.002s
I20260321 02:58:19.727362 11430 consensus_queue.cc:1048] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:19.728997 11424 tablet_bootstrap.cc:492] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:19.734782 11425 tablet_bootstrap.cc:654] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.743620 11429 tablet_bootstrap.cc:492] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:19.744279 11429 ts_tablet_manager.cc:1403] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.110s user 0.006s sys 0.011s
I20260321 02:58:19.753046 11430 consensus_queue.cc:1048] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:19.754099 11424 tablet_bootstrap.cc:654] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.761126 11425 tablet_bootstrap.cc:492] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:19.761592 11425 ts_tablet_manager.cc:1403] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.046s user 0.012s sys 0.019s
I20260321 02:58:19.764339 11425 raft_consensus.cc:359] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.765043 11425 raft_consensus.cc:385] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.765323 11425 raft_consensus.cc:740] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.765873 11429 raft_consensus.cc:359] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.774890 11429 raft_consensus.cc:385] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.775166 11429 raft_consensus.cc:740] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.775779 11429 consensus_queue.cc:260] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.778194 11429 ts_tablet_manager.cc:1434] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.034s user 0.006s sys 0.000s
I20260321 02:58:19.779089 11429 tablet_bootstrap.cc:492] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:19.766865 11428 consensus_queue.cc:1048] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260321 02:58:19.766153 11425 consensus_queue.cc:260] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.785423 11424 tablet_bootstrap.cc:492] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:19.785847 11424 ts_tablet_manager.cc:1403] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.021s
I20260321 02:58:19.786212 11425 ts_tablet_manager.cc:1434] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20260321 02:58:19.786800 11429 tablet_bootstrap.cc:654] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.789045 11424 raft_consensus.cc:359] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.789754 11424 raft_consensus.cc:385] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.790012 11424 raft_consensus.cc:740] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.790634 11424 consensus_queue.cc:260] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.791917 11425 tablet_bootstrap.cc:492] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:19.796293 11492 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:19.796941 11492 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.795658 11444 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:19.800961 11444 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.802206 11425 tablet_bootstrap.cc:654] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.803758 11424 ts_tablet_manager.cc:1434] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.018s user 0.002s sys 0.005s
I20260321 02:58:19.805805 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:19.806944 11235 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
I20260321 02:58:19.808883 11487 consensus_queue.cc:1048] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:19.811550 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:19.813086 11275 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:19.814281 11442 raft_consensus.cc:2804] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:19.814760 11442 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:19.815196 11442 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.816725 11424 tablet_bootstrap.cc:492] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:19.820921 11428 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:19.821642 11428 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.824198 11428 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:19.829762 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:19.830729 11385 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
I20260321 02:58:19.831086 11424 tablet_bootstrap.cc:654] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.832757 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:19.833652 11384 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:19.803555 11444 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:19.799654 11492 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:19.841804 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:19.846874 11200 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:19.854044 11492 raft_consensus.cc:2804] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:19.855072 11492 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:19.855557 11492 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.856319 11424 tablet_bootstrap.cc:492] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:19.857034 11424 ts_tablet_manager.cc:1403] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.014s
I20260321 02:58:19.858796 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:19.859823 11235 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
W20260321 02:58:19.858764 11200 leader_election.cc:343] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:19.867213 11349 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:19.868628 11428 raft_consensus.cc:2804] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:19.869509 11428 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:19.870011 11428 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.873422 11424 raft_consensus.cc:359] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.880618 11458 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:19.885426 11424 raft_consensus.cc:385] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.886133 11424 raft_consensus.cc:740] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.885682 11458 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.887017 11424 consensus_queue.cc:260] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.889393 11492 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.889974 11424 ts_tablet_manager.cc:1434] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.032s user 0.006s sys 0.001s
I20260321 02:58:19.891431 11424 tablet_bootstrap.cc:492] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:19.896585 11458 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:19.891323 11492 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:19.907352 11424 tablet_bootstrap.cc:654] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:19.884549 11442 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.905277 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:58:19.935354 11349 leader_election.cc:343] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:19.935837 11424 tablet_bootstrap.cc:492] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:19.937294 11424 ts_tablet_manager.cc:1403] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.046s user 0.010s sys 0.004s
I20260321 02:58:19.939863 11311 raft_consensus.cc:2393] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 1: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:19.940393 11424 raft_consensus.cc:359] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.941250 11424 raft_consensus.cc:385] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:19.941617 11424 raft_consensus.cc:740] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:19.942473 11424 consensus_queue.cc:260] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.935884 11458 consensus_queue.cc:1048] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260321 02:58:19.946132 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:19.947027 11235 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:19.947500 11424 ts_tablet_manager.cc:1434] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260321 02:58:19.951531 11424 tablet_bootstrap.cc:492] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:19.962916 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:19.964000 11442 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:19.974356 11428 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:19.977615 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:19.982877 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:19.983718 11384 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.005496 11385 raft_consensus.cc:2393] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 1: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:58:20.012617 11428 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:20.017328 11424 tablet_bootstrap.cc:654] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.017174 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:20.043720 11309 raft_consensus.cc:2393] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 1: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:20.044108 11429 tablet_bootstrap.cc:492] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:20.020211 11384 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:20.022053 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:20.044817 11234 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:20.024035 11235 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:20.025499 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:58:20.036640 11200 leader_election.cc:343] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:20.042148 11444 consensus_queue.cc:1048] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:20.048733 11349 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:20.049612 11200 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:20.050140 11349 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:20.050432 11444 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:20.050166 11275 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:20.051175 11458 raft_consensus.cc:2804] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:20.050973 11444 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.051553 11458 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:20.051918 11458 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.053195 11429 ts_tablet_manager.cc:1403] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.274s user 0.120s sys 0.134s
I20260321 02:58:20.053133 11444 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:20.053858 11459 raft_consensus.cc:2749] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260321 02:58:20.054489 11446 raft_consensus.cc:2804] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:20.054993 11446 raft_consensus.cc:697] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:20.052531 11442 raft_consensus.cc:2804] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:20.056272 11429 raft_consensus.cc:359] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.057000 11442 raft_consensus.cc:697] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:58:20.018123 11492 consensus_queue.cc:1048] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:20.059255 11442 consensus_queue.cc:237] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.042708 11425 tablet_bootstrap.cc:492] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:20.062196 11425 ts_tablet_manager.cc:1403] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.270s user 0.072s sys 0.145s
I20260321 02:58:20.064764 11446 consensus_queue.cc:237] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.065672 11309 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.061388 11458 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.065814 11425 raft_consensus.cc:359] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.068322 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:20.069084 11385 raft_consensus.cc:2468] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
I20260321 02:58:20.070025 11425 raft_consensus.cc:385] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.070871 11425 raft_consensus.cc:740] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.071794 11458 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:20.072796 11275 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:20.073966 11444 raft_consensus.cc:2804] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:20.074333 11444 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:20.074842 11444 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.076262 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:20.077914 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:58:20.078550 11349 leader_election.cc:343] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:20.057071 11429 raft_consensus.cc:385] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.083535 11429 raft_consensus.cc:740] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.084301 11429 consensus_queue.cc:260] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.085181 11444 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.072824 11425 consensus_queue.cc:260] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.088629 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:20.091018 11425 ts_tablet_manager.cc:1434] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260321 02:58:20.093499 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:20.094285 11233 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.095721 11429 ts_tablet_manager.cc:1434] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.042s user 0.008s sys 0.000s
I20260321 02:58:20.096712 11429 tablet_bootstrap.cc:492] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:20.079032 11235 raft_consensus.cc:2468] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
I20260321 02:58:20.099825 11425 tablet_bootstrap.cc:492] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:20.100996 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:20.104146 11384 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.102548 11133 catalog_manager.cc:5671] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:20.112231 11446 consensus_queue.cc:1048] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260321 02:58:20.119267 11424 tablet_bootstrap.cc:492] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:20.119791 11424 ts_tablet_manager.cc:1403] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.169s user 0.012s sys 0.002s
I20260321 02:58:20.102981 11444 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:20.123601 11425 tablet_bootstrap.cc:654] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.142040 11429 tablet_bootstrap.cc:654] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.146772 11424 raft_consensus.cc:359] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.147524 11424 raft_consensus.cc:385] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.148818 11487 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:20.149351 11487 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.151826 11487 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:20.154224 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:20.155474 11384 raft_consensus.cc:2468] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:20.156100 11424 raft_consensus.cc:740] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.156991 11424 consensus_queue.cc:260] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.155927 11444 consensus_queue.cc:1048] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260321 02:58:20.155124 11458 consensus_queue.cc:1048] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
W20260321 02:58:20.159546 11200 leader_election.cc:343] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:20.154657 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:20.142624 11233 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:20.162212 11458 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:58:20.163350 11200 leader_election.cc:343] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:20.163688 11458 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.163923 11200 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:20.168516 11458 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:20.170102 11275 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:20.171693 11444 raft_consensus.cc:2804] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:20.172407 11444 raft_consensus.cc:697] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:58:20.176196 11424 ts_tablet_manager.cc:1434] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.056s user 0.009s sys 0.000s
I20260321 02:58:20.177601 11424 tablet_bootstrap.cc:492] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:20.184926 11531 raft_consensus.cc:2749] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:20.186556 11444 consensus_queue.cc:237] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.198324 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:20.198793 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:20.199436 11233 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:20.174623 11130 catalog_manager.cc:5671] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:20.198355 11349 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:20.202309 11458 raft_consensus.cc:2804] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
W20260321 02:58:20.210474 11349 leader_election.cc:343] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:20.212666 11349 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:20.214485 11428 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:20.214924 11428 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:20.216490 11428 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.204169 11458 raft_consensus.cc:697] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:20.217717 11458 consensus_queue.cc:237] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.225592 11459 consensus_queue.cc:1048] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260321 02:58:20.237756 11459 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:20.240556 11459 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.246089 11459 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:20.249866 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:20.247570 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:20.250355 11425 tablet_bootstrap.cc:492] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:20.250801 11425 ts_tablet_manager.cc:1403] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.151s user 0.015s sys 0.000s
W20260321 02:58:20.251993 11349 leader_election.cc:343] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:20.254791 11429 tablet_bootstrap.cc:492] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:20.255218 11429 ts_tablet_manager.cc:1403] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.159s user 0.010s sys 0.014s
W20260321 02:58:20.260776 11349 leader_election.cc:343] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:20.264302 11424 tablet_bootstrap.cc:654] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.264387 11349 leader_election.cc:304] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:20.264752 11429 raft_consensus.cc:359] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.265413 11429 raft_consensus.cc:385] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.265704 11429 raft_consensus.cc:740] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.249830 11428 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.265902 11425 raft_consensus.cc:359] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.266829 11429 consensus_queue.cc:260] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.268045 11425 raft_consensus.cc:385] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.268694 11425 raft_consensus.cc:740] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.267890 11459 raft_consensus.cc:2749] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:20.272317 11429 ts_tablet_manager.cc:1434] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.016s user 0.005s sys 0.001s
I20260321 02:58:20.273545 11429 tablet_bootstrap.cc:492] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:20.273312 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:20.274168 11234 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.275904 11425 consensus_queue.cc:260] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.279119 11425 ts_tablet_manager.cc:1434] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.028s user 0.004s sys 0.004s
I20260321 02:58:20.289532 11425 tablet_bootstrap.cc:492] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:20.288793 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:20.287607 11428 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:20.292304 11311 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.295877 11486 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:20.296675 11486 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.299829 11486 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:20.300637 11429 tablet_bootstrap.cc:654] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.300243 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:58:20.304924 11275 leader_election.cc:343] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:20.303180 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:20.304605 11311 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
W20260321 02:58:20.308035 11275 leader_election.cc:343] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:20.308717 11275 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:20.310046 11486 raft_consensus.cc:2749] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:20.311710 11349 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:20.312819 11459 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:20.313303 11459 raft_consensus.cc:697] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:20.314677 11459 consensus_queue.cc:237] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.325312 11234 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:20.333719 11424 tablet_bootstrap.cc:492] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:20.334441 11424 ts_tablet_manager.cc:1403] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.157s user 0.016s sys 0.000s
I20260321 02:58:20.339524 11424 raft_consensus.cc:359] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.342005 11424 raft_consensus.cc:385] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.342401 11424 raft_consensus.cc:740] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.348421 11424 consensus_queue.cc:260] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.352430 11429 tablet_bootstrap.cc:492] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:20.352905 11429 ts_tablet_manager.cc:1403] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.080s user 0.018s sys 0.004s
I20260321 02:58:20.354754 11424 ts_tablet_manager.cc:1434] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.020s user 0.005s sys 0.004s
I20260321 02:58:20.355683 11424 tablet_bootstrap.cc:492] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:20.356868 11425 tablet_bootstrap.cc:654] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.362874 11130 catalog_manager.cc:5671] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:20.368119 11429 raft_consensus.cc:359] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.368938 11429 raft_consensus.cc:385] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.369760 11429 raft_consensus.cc:740] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.370512 11429 consensus_queue.cc:260] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.365687 11130 catalog_manager.cc:5671] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:20.373252 11424 tablet_bootstrap.cc:654] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.375942 11429 ts_tablet_manager.cc:1434] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.023s user 0.003s sys 0.004s
I20260321 02:58:20.385167 11424 tablet_bootstrap.cc:492] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:20.385859 11424 ts_tablet_manager.cc:1403] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.007s
I20260321 02:58:20.387226 11429 tablet_bootstrap.cc:492] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:20.389371 11424 raft_consensus.cc:359] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.390173 11424 raft_consensus.cc:385] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.390470 11424 raft_consensus.cc:740] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.391161 11424 consensus_queue.cc:260] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.393615 11424 ts_tablet_manager.cc:1434] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260321 02:58:20.394497 11424 tablet_bootstrap.cc:492] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:20.403409 11425 tablet_bootstrap.cc:492] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:20.404029 11425 ts_tablet_manager.cc:1403] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.124s user 0.023s sys 0.017s
I20260321 02:58:20.404902 11429 tablet_bootstrap.cc:654] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.407073 11425 raft_consensus.cc:359] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.408069 11425 raft_consensus.cc:385] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.408435 11425 raft_consensus.cc:740] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.409528 11424 tablet_bootstrap.cc:654] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.410208 11425 consensus_queue.cc:260] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.416756 11425 ts_tablet_manager.cc:1434] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260321 02:58:20.417779 11425 tablet_bootstrap.cc:492] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:20.418140 11429 tablet_bootstrap.cc:492] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:20.418859 11429 ts_tablet_manager.cc:1403] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.007s
I20260321 02:58:20.418367 11133 catalog_manager.cc:5671] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:20.427387 11429 raft_consensus.cc:359] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.428387 11429 raft_consensus.cc:385] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.428706 11429 raft_consensus.cc:740] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.429780 11429 consensus_queue.cc:260] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.434478 11429 ts_tablet_manager.cc:1434] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.015s user 0.007s sys 0.005s
I20260321 02:58:20.435459 11429 tablet_bootstrap.cc:492] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:20.435667 11425 tablet_bootstrap.cc:654] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.435509 11459 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:20.436352 11459 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:20.437502 11182 auto_rebalancer.cc:247] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260321 02:58:20.438418 11459 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:20.441110 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:20.441985 11429 tablet_bootstrap.cc:654] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.441138 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:58:20.443534 11349 leader_election.cc:343] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
W20260321 02:58:20.444329 11349 leader_election.cc:343] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:20.444756 11349 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:20.445674 11424 tablet_bootstrap.cc:492] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:20.446169 11424 ts_tablet_manager.cc:1403] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.016s
I20260321 02:58:20.448312 11459 raft_consensus.cc:2749] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:20.450757 11425 tablet_bootstrap.cc:492] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:20.451687 11425 ts_tablet_manager.cc:1403] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.005s
I20260321 02:58:20.454573 11425 raft_consensus.cc:359] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.455344 11425 raft_consensus.cc:385] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.455616 11425 raft_consensus.cc:740] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.455902 11424 raft_consensus.cc:359] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.456647 11424 raft_consensus.cc:385] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.456346 11425 consensus_queue.cc:260] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.456909 11424 raft_consensus.cc:740] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.461365 11424 consensus_queue.cc:260] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.463584 11424 ts_tablet_manager.cc:1434] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.017s user 0.002s sys 0.005s
I20260321 02:58:20.463584 11425 ts_tablet_manager.cc:1434] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260321 02:58:20.464710 11424 tablet_bootstrap.cc:492] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:20.464773 11425 tablet_bootstrap.cc:492] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:20.471246 11425 tablet_bootstrap.cc:654] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.471246 11424 tablet_bootstrap.cc:654] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.474148 11428 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:20.474457 11446 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:20.474710 11428 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.475241 11446 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.477154 11446 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:20.477149 11428 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:20.478892 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:20.479220 11429 tablet_bootstrap.cc:492] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:20.479120 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:20.479542 11311 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:20.479696 11429 ts_tablet_manager.cc:1403] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.044s user 0.016s sys 0.004s
I20260321 02:58:20.482885 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:20.479756 11234 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:20.486225 11349 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:20.480556 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:58:20.489725 11200 leader_election.cc:343] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
W20260321 02:58:20.490422 11200 leader_election.cc:343] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:20.490396 11429 raft_consensus.cc:359] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.491044 11200 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:20.491123 11459 raft_consensus.cc:2804] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:20.492408 11459 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:20.492795 11459 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.491186 11429 raft_consensus.cc:385] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.493453 11429 raft_consensus.cc:740] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.494290 11424 tablet_bootstrap.cc:492] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:20.494820 11429 consensus_queue.cc:260] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.495275 11424 ts_tablet_manager.cc:1403] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.009s
I20260321 02:58:20.497287 11446 raft_consensus.cc:2749] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:20.498034 11429 ts_tablet_manager.cc:1434] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.011s user 0.004s sys 0.004s
I20260321 02:58:20.500484 11425 tablet_bootstrap.cc:492] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:20.500921 11425 ts_tablet_manager.cc:1403] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.005s
I20260321 02:58:20.504515 11459 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.506848 11424 raft_consensus.cc:359] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.507241 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:20.507552 11424 raft_consensus.cc:385] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.507781 11234 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.507840 11424 raft_consensus.cc:740] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.508091 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:20.508760 11310 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.508562 11424 consensus_queue.cc:260] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.510038 11425 raft_consensus.cc:359] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.510741 11425 raft_consensus.cc:385] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.511030 11425 raft_consensus.cc:740] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.511726 11425 consensus_queue.cc:260] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.513662 11428 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:20.514431 11424 ts_tablet_manager.cc:1434] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.019s user 0.003s sys 0.003s
I20260321 02:58:20.514166 11428 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.515396 11424 tablet_bootstrap.cc:492] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:20.515827 11234 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:20.516268 11310 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:20.516846 11429 tablet_bootstrap.cc:492] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:20.517026 11459 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:20.518016 11349 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:20.516263 11428 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:20.521026 11428 raft_consensus.cc:2804] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:20.521500 11428 raft_consensus.cc:697] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:20.522306 11428 consensus_queue.cc:237] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.525308 11424 tablet_bootstrap.cc:654] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.525444 11429 tablet_bootstrap.cc:654] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.520251 11425 ts_tablet_manager.cc:1434] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20260321 02:58:20.526558 11446 consensus_queue.cc:1048] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:20.527371 11425 tablet_bootstrap.cc:492] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:20.533250 11486 consensus_queue.cc:1048] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:20.531780 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:20.542498 11234 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:20.544152 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:20.544034 11349 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:20.549356 11553 raft_consensus.cc:2804] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:20.549803 11553 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:20.552474 11553 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
W20260321 02:58:20.548880 11349 leader_election.cc:343] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:20.563736 11540 consensus_queue.cc:1048] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260321 02:58:20.584918 11459 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:20.585736 11459 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.589146 11459 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:20.588910 11553 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.591245 11553 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:20.596138 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:20.596871 11310 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:20.596151 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:20.600221 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:20.602162 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:20.602783 11233 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.609965 11310 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.614337 11425 tablet_bootstrap.cc:654] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.616070 11429 tablet_bootstrap.cc:492] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:20.616700 11429 ts_tablet_manager.cc:1403] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.116s user 0.010s sys 0.006s
I20260321 02:58:20.620100 11424 tablet_bootstrap.cc:492] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:20.620743 11424 ts_tablet_manager.cc:1403] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.106s user 0.016s sys 0.007s
I20260321 02:58:20.623670 11424 raft_consensus.cc:359] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.624437 11424 raft_consensus.cc:385] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.625054 11424 raft_consensus.cc:740] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.625839 11424 consensus_queue.cc:260] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:20.628604 11349 leader_election.cc:343] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:20.628067 11133 catalog_manager.cc:5671] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:20.630226 11349 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:20.632083 11540 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:58:20.637164 11349 leader_election.cc:343] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:20.636826 11540 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.632598 11446 consensus_queue.cc:1048] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:20.629316 11233 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:20.640691 11553 raft_consensus.cc:2804] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:20.641144 11553 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:20.633390 11425 tablet_bootstrap.cc:492] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:20.641759 11553 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.644871 11446 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:20.640463 11540 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:20.645360 11446 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.647464 11446 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:20.648897 11425 ts_tablet_manager.cc:1403] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.122s user 0.013s sys 0.000s
I20260321 02:58:20.651037 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:20.651846 11233 raft_consensus.cc:2393] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 1: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:58:20.651502 11425 raft_consensus.cc:359] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.652187 11425 raft_consensus.cc:385] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.652460 11425 raft_consensus.cc:740] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.636072 11424 ts_tablet_manager.cc:1434] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.015s user 0.009s sys 0.000s
I20260321 02:58:20.654708 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:20.657087 11385 raft_consensus.cc:2468] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:20.654733 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:20.655601 11429 raft_consensus.cc:359] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.660041 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:20.659838 11429 raft_consensus.cc:385] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.660801 11429 raft_consensus.cc:740] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
W20260321 02:58:20.661374 11200 leader_election.cc:343] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:20.661454 11429 consensus_queue.cc:260] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.662217 11200 leader_election.cc:304] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:20.663898 11446 raft_consensus.cc:2804] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:20.664299 11446 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:20.664673 11446 raft_consensus.cc:3060] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.668941 11424 tablet_bootstrap.cc:492] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:20.670873 11349 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:20.672446 11459 raft_consensus.cc:2804] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:20.673202 11459 raft_consensus.cc:697] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:20.674093 11459 consensus_queue.cc:237] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.675988 11311 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.684123 11233 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.684182 11553 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.686519 11553 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:20.679575 11429 ts_tablet_manager.cc:1434] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.063s user 0.007s sys 0.000s
I20260321 02:58:20.681773 11425 consensus_queue.cc:260] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.688954 11429 tablet_bootstrap.cc:492] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:20.690115 11425 ts_tablet_manager.cc:1434] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20260321 02:58:20.691063 11425 tablet_bootstrap.cc:492] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:20.683112 11275 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:20.692350 11428 consensus_queue.cc:1048] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:20.694132 11446 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.695083 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:58:20.697530 11349 leader_election.cc:343] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:20.697530 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:20.705667 11429 tablet_bootstrap.cc:654] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.708808 11428 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:20.709357 11428 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.711882 11428 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:58:20.713973 11200 leader_election.cc:343] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:20.716704 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:20.717478 11234 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:20.720320 11349 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:20.721395 11428 raft_consensus.cc:2804] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:20.721837 11428 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:20.722234 11428 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.733337 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:20.734216 11309 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.748440 11424 tablet_bootstrap.cc:654] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.745761 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:20.752426 11130 catalog_manager.cc:5671] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:20.754489 11446 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:20.761982 11540 consensus_queue.cc:1048] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:20.756188 11442 raft_consensus.cc:2749] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:20.764878 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:20.765592 11383 raft_consensus.cc:3060] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
W20260321 02:58:20.766641 11349 leader_election.cc:343] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:20.770267 11428 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.771677 11442 consensus_queue.cc:1048] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:20.772377 11428 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:20.775369 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:20.776214 11235 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.777238 11429 tablet_bootstrap.cc:492] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:20.777863 11429 ts_tablet_manager.cc:1403] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.089s user 0.020s sys 0.019s
I20260321 02:58:20.780042 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:20.781131 11383 raft_consensus.cc:2468] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
W20260321 02:58:20.786062 11349 leader_election.cc:343] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:20.786693 11235 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:20.786827 11428 consensus_queue.cc:1048] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:20.805243 11309 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:20.806427 11429 raft_consensus.cc:359] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.807236 11429 raft_consensus.cc:385] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.807929 11429 raft_consensus.cc:740] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.808518 11200 leader_election.cc:304] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:20.809118 11429 consensus_queue.cc:260] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.809932 11531 raft_consensus.cc:2804] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:20.810418 11531 raft_consensus.cc:697] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:20.811666 11429 ts_tablet_manager.cc:1434] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.033s user 0.006s sys 0.001s
I20260321 02:58:20.812374 11424 tablet_bootstrap.cc:492] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:20.811225 11531 consensus_queue.cc:237] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.812875 11424 ts_tablet_manager.cc:1403] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.144s user 0.006s sys 0.009s
I20260321 02:58:20.844218 11349 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:20.816272 11429 tablet_bootstrap.cc:492] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:20.851903 11425 tablet_bootstrap.cc:654] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.857152 11349 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:20.860144 11553 raft_consensus.cc:2804] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:20.860481 11459 consensus_queue.cc:1048] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:20.860950 11549 raft_consensus.cc:2804] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:20.861545 11549 raft_consensus.cc:697] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:20.860685 11553 raft_consensus.cc:697] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:20.862447 11549 consensus_queue.cc:237] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.863636 11553 consensus_queue.cc:237] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.866758 11424 raft_consensus.cc:359] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.867482 11424 raft_consensus.cc:385] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.867743 11424 raft_consensus.cc:740] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.868474 11424 consensus_queue.cc:260] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.869261 11429 tablet_bootstrap.cc:654] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.871054 11424 ts_tablet_manager.cc:1434] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.016s user 0.003s sys 0.003s
I20260321 02:58:20.872255 11424 tablet_bootstrap.cc:492] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:20.875533 11428 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:20.876276 11428 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.878970 11428 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:20.881049 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:20.882105 11235 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:20.891036 11424 tablet_bootstrap.cc:654] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.901672 11425 tablet_bootstrap.cc:492] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:20.902206 11425 ts_tablet_manager.cc:1403] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.211s user 0.016s sys 0.000s
I20260321 02:58:20.901422 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:20.913444 11349 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:20.914431 11582 raft_consensus.cc:2804] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:20.914811 11582 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:20.907564 11442 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:20.910830 11425 raft_consensus.cc:359] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.916513 11425 raft_consensus.cc:385] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.916388 11442 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.916867 11425 raft_consensus.cc:740] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.917634 11425 consensus_queue.cc:260] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.915155 11582 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.918413 11442 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:20.920092 11425 ts_tablet_manager.cc:1434] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260321 02:58:20.906517 11130 catalog_manager.cc:5671] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
W20260321 02:58:20.921948 11349 leader_election.cc:343] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:20.922156 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:20.923452 11235 raft_consensus.cc:2393] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 1: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:58:20.925452 11429 tablet_bootstrap.cc:492] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:20.925799 11425 tablet_bootstrap.cc:492] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:20.925936 11429 ts_tablet_manager.cc:1403] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.113s user 0.022s sys 0.021s
I20260321 02:58:20.926141 11549 consensus_queue.cc:1048] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:20.930042 11429 raft_consensus.cc:359] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.930742 11429 raft_consensus.cc:385] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.931013 11429 raft_consensus.cc:740] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.930881 11133 catalog_manager.cc:5671] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:20.932674 11429 consensus_queue.cc:260] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.932554 11133 catalog_manager.cc:5671] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:20.935006 11429 ts_tablet_manager.cc:1434] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.009s user 0.005s sys 0.005s
I20260321 02:58:20.935732 11429 tablet_bootstrap.cc:492] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:20.941250 11424 tablet_bootstrap.cc:492] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:20.941753 11424 ts_tablet_manager.cc:1403] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.070s user 0.022s sys 0.011s
I20260321 02:58:20.940871 11582 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.946605 11549 consensus_queue.cc:1048] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:20.946455 11582 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:20.951617 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:20.952284 11311 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.952350 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:20.953603 11424 raft_consensus.cc:359] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.954253 11424 raft_consensus.cc:385] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:20.954510 11424 raft_consensus.cc:740] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:20.958359 11487 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:20.958858 11487 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.960691 11487 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:20.961179 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:20.961447 11311 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:20.961771 11235 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.961807 11424 consensus_queue.cc:260] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.963088 11425 tablet_bootstrap.cc:654] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.963182 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:20.963831 11309 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:20.964151 11429 tablet_bootstrap.cc:654] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.965153 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:20.965770 11385 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:20.966921 11200 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:20.968271 11487 raft_consensus.cc:2804] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:20.968667 11487 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:20.968971 11487 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.969163 11424 ts_tablet_manager.cc:1434] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.027s user 0.005s sys 0.001s
I20260321 02:58:20.970513 11424 tablet_bootstrap.cc:492] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:20.982162 11424 tablet_bootstrap.cc:654] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:20.991847 11487 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:20.995397 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:20.996057 11309 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:20.996497 11424 tablet_bootstrap.cc:492] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:20.997155 11424 ts_tablet_manager.cc:1403] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.003s
I20260321 02:58:21.001098 11424 raft_consensus.cc:359] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.002094 11424 raft_consensus.cc:385] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:21.002607 11424 raft_consensus.cc:740] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:21.007691 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:21.008363 11385 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:21.009337 11235 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:21.003572 11424 consensus_queue.cc:260] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.015753 11424 ts_tablet_manager.cc:1434] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20260321 02:58:21.017196 11424 tablet_bootstrap.cc:492] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:21.004668 11309 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:21.020413 11275 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:21.021590 11442 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:21.021801 11200 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:21.023128 11531 raft_consensus.cc:2804] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:21.023751 11531 raft_consensus.cc:697] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:21.024631 11531 consensus_queue.cc:237] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.028110 11349 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:21.030207 11549 raft_consensus.cc:2804] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:21.030875 11549 raft_consensus.cc:697] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:21.032452 11549 consensus_queue.cc:237] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.034090 11424 tablet_bootstrap.cc:654] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:21.036837 11487 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:21.037595 11442 raft_consensus.cc:2749] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:21.020464 11385 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:21.042125 11429 tablet_bootstrap.cc:492] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:21.043249 11429 ts_tablet_manager.cc:1403] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.108s user 0.018s sys 0.024s
I20260321 02:58:21.046383 11429 raft_consensus.cc:359] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.047163 11429 raft_consensus.cc:385] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:21.047554 11429 raft_consensus.cc:740] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:21.048432 11429 consensus_queue.cc:260] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.052186 11424 tablet_bootstrap.cc:492] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:21.052879 11424 ts_tablet_manager.cc:1403] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.036s user 0.013s sys 0.001s
I20260321 02:58:21.056335 11425 tablet_bootstrap.cc:492] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:21.056871 11425 ts_tablet_manager.cc:1403] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.136s user 0.030s sys 0.045s
I20260321 02:58:21.057389 11429 ts_tablet_manager.cc:1434] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.014s user 0.003s sys 0.003s
I20260321 02:58:21.058326 11429 tablet_bootstrap.cc:492] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:21.058871 11531 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:21.059329 11531 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.062595 11531 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:21.063090 11549 consensus_queue.cc:1048] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:21.064543 11425 raft_consensus.cc:359] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.065220 11425 raft_consensus.cc:385] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:21.065263 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:21.065361 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:21.066138 11429 tablet_bootstrap.cc:654] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:21.065950 11424 raft_consensus.cc:359] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.066184 11309 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:21.066569 11424 raft_consensus.cc:385] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:21.066884 11424 raft_consensus.cc:740] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:21.065487 11425 raft_consensus.cc:740] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:21.068416 11200 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:21.069460 11531 raft_consensus.cc:2804] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:21.069958 11531 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:21.070377 11531 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:21.067507 11424 consensus_queue.cc:260] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.066184 11385 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:21.068444 11425 consensus_queue.cc:260] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.075840 11133 catalog_manager.cc:5671] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:21.079723 11425 ts_tablet_manager.cc:1434] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.023s user 0.005s sys 0.002s
I20260321 02:58:21.080897 11425 tablet_bootstrap.cc:492] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:21.081220 11531 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.081929 11132 catalog_manager.cc:5671] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:21.088203 11531 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:21.090749 11429 tablet_bootstrap.cc:492] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:21.090678 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:21.092288 11429 ts_tablet_manager.cc:1403] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.013s
I20260321 02:58:21.092433 11385 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:21.093168 11424 ts_tablet_manager.cc:1434] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.040s user 0.009s sys 0.000s
I20260321 02:58:21.093039 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:21.094095 11424 tablet_bootstrap.cc:492] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:21.094164 11311 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:21.103907 11429 raft_consensus.cc:359] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.104972 11429 raft_consensus.cc:385] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:21.105253 11429 raft_consensus.cc:740] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:21.108989 11424 tablet_bootstrap.cc:654] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:21.106009 11429 consensus_queue.cc:260] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.124632 11311 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:21.126292 11200 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:21.128147 11531 raft_consensus.cc:2804] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:21.132272 11429 ts_tablet_manager.cc:1434] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.040s user 0.003s sys 0.006s
I20260321 02:58:21.128753 11531 raft_consensus.cc:697] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:21.133285 11429 tablet_bootstrap.cc:492] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:21.133638 11531 consensus_queue.cc:237] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.129750 11385 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:21.140887 11425 tablet_bootstrap.cc:654] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:21.146476 11459 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:21.146942 11429 tablet_bootstrap.cc:654] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:21.147275 11459 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.153043 11549 consensus_queue.cc:1048] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:21.153894 11424 tablet_bootstrap.cc:492] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:21.154428 11424 ts_tablet_manager.cc:1403] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.061s user 0.014s sys 0.003s
I20260321 02:58:21.155247 11459 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:21.164292 11487 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:21.164835 11487 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.167263 11487 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:21.167132 11424 raft_consensus.cc:359] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.167930 11424 raft_consensus.cc:385] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:21.168252 11424 raft_consensus.cc:740] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:21.168937 11424 consensus_queue.cc:260] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.170436 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:21.171277 11311 raft_consensus.cc:2393] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 1: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:58:21.171578 11424 ts_tablet_manager.cc:1434] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.017s user 0.004s sys 0.001s
I20260321 02:58:21.172880 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:21.175340 11200 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:21.176009 11132 catalog_manager.cc:5671] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:21.177611 11424 tablet_bootstrap.cc:492] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:21.177510 11487 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:21.197103 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:21.197604 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:21.210055 11425 tablet_bootstrap.cc:492] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:21.210554 11425 ts_tablet_manager.cc:1403] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.130s user 0.017s sys 0.018s
W20260321 02:58:21.216310 11349 leader_election.cc:343] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:21.216553 11424 tablet_bootstrap.cc:654] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:21.212404 11487 raft_consensus.cc:2749] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:21.217559 11429 tablet_bootstrap.cc:492] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:21.218497 11429 ts_tablet_manager.cc:1403] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.085s user 0.017s sys 0.020s
I20260321 02:58:21.220504 11459 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:21.221001 11459 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.222204 11425 raft_consensus.cc:359] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.222891 11425 raft_consensus.cc:385] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:21.223160 11425 raft_consensus.cc:740] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:21.223764 11425 consensus_queue.cc:260] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.226094 11429 raft_consensus.cc:359] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.226573 11459 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:21.227221 11425 ts_tablet_manager.cc:1434] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.016s user 0.006s sys 0.002s
I20260321 02:58:21.228183 11425 tablet_bootstrap.cc:492] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:21.226815 11429 raft_consensus.cc:385] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:21.229888 11429 raft_consensus.cc:740] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:21.230785 11429 consensus_queue.cc:260] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.234223 11429 ts_tablet_manager.cc:1434] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.015s user 0.004s sys 0.003s
I20260321 02:58:21.235059 11429 tablet_bootstrap.cc:492] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:21.251201 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:21.256973 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:21.260363 11425 tablet_bootstrap.cc:654] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
W20260321 02:58:21.261929 11349 leader_election.cc:343] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:21.262416 11349 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:21.268144 11459 raft_consensus.cc:2749] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:21.269502 11459 consensus_queue.cc:1048] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260321 02:58:21.271168 11349 leader_election.cc:343] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:21.278996 11429 tablet_bootstrap.cc:654] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
W20260321 02:58:21.283604 11349 leader_election.cc:343] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:21.285907 11424 tablet_bootstrap.cc:492] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:21.286406 11424 ts_tablet_manager.cc:1403] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.109s user 0.027s sys 0.034s
I20260321 02:58:21.286028 11349 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:21.287184 11428 raft_consensus.cc:2749] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:21.289220 11424 raft_consensus.cc:359] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.290035 11424 raft_consensus.cc:385] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:21.290293 11424 raft_consensus.cc:740] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:21.290979 11424 consensus_queue.cc:260] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.293139 11424 ts_tablet_manager.cc:1434] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260321 02:58:21.296959 11425 tablet_bootstrap.cc:492] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:21.297633 11425 ts_tablet_manager.cc:1403] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.070s user 0.014s sys 0.004s
I20260321 02:58:21.297796 11424 tablet_bootstrap.cc:492] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:21.300495 11429 tablet_bootstrap.cc:492] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:21.301301 11425 raft_consensus.cc:359] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.302181 11425 raft_consensus.cc:385] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:21.302443 11425 raft_consensus.cc:740] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:21.303040 11425 consensus_queue.cc:260] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.305223 11424 tablet_bootstrap.cc:654] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:21.305567 11425 ts_tablet_manager.cc:1434] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260321 02:58:21.306267 11429 ts_tablet_manager.cc:1403] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.071s user 0.008s sys 0.007s
I20260321 02:58:21.309355 11429 raft_consensus.cc:359] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.310037 11429 raft_consensus.cc:385] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:21.310233 11429 raft_consensus.cc:740] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:21.310719 11429 consensus_queue.cc:260] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.312434 11425 tablet_bootstrap.cc:492] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:21.317940 11429 ts_tablet_manager.cc:1434] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260321 02:58:21.318933 11429 tablet_bootstrap.cc:492] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:21.318956 11425 tablet_bootstrap.cc:654] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:21.321305 11424 tablet_bootstrap.cc:492] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:21.321720 11424 ts_tablet_manager.cc:1403] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.008s
I20260321 02:58:21.324115 11424 raft_consensus.cc:359] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.324867 11424 raft_consensus.cc:385] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:21.325115 11424 raft_consensus.cc:740] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:21.325688 11424 consensus_queue.cc:260] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.327699 11424 ts_tablet_manager.cc:1434] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260321 02:58:21.328590 11424 tablet_bootstrap.cc:492] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:21.328925 11425 tablet_bootstrap.cc:492] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:21.329389 11425 ts_tablet_manager.cc:1403] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.004s
I20260321 02:58:21.331738 11425 raft_consensus.cc:359] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.332484 11425 raft_consensus.cc:385] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:21.332749 11425 raft_consensus.cc:740] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:21.333335 11425 consensus_queue.cc:260] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.334813 11429 tablet_bootstrap.cc:654] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:21.334813 11424 tablet_bootstrap.cc:654] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:21.335673 11425 ts_tablet_manager.cc:1434] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260321 02:58:21.336735 11537 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:21.337195 11537 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.339157 11537 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:21.342175 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:21.342806 11311 raft_consensus.cc:3060] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:21.343132 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:21.343430 11425 tablet_bootstrap.cc:492] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
W20260321 02:58:21.344746 11275 leader_election.cc:343] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
W20260321 02:58:21.345371 11275 leader_election.cc:343] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:21.346263 11275 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:21.350225 11429 tablet_bootstrap.cc:492] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:21.350683 11429 ts_tablet_manager.cc:1403] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.007s
I20260321 02:58:21.351480 11425 tablet_bootstrap.cc:654] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:21.353400 11531 consensus_queue.cc:1048] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:21.353992 11429 raft_consensus.cc:359] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.354893 11429 raft_consensus.cc:385] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:21.355229 11429 raft_consensus.cc:740] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:21.357606 11429 consensus_queue.cc:260] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.358596 11540 raft_consensus.cc:2749] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:21.358990 11424 tablet_bootstrap.cc:492] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:21.359475 11424 ts_tablet_manager.cc:1403] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.012s
I20260321 02:58:21.360806 11429 ts_tablet_manager.cc:1434] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260321 02:58:21.361805 11429 tablet_bootstrap.cc:492] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:21.368453 11429 tablet_bootstrap.cc:654] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:21.370980 11424 raft_consensus.cc:359] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.374446 11424 raft_consensus.cc:385] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:21.375030 11424 raft_consensus.cc:740] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:21.383150 11553 consensus_queue.cc:1048] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:21.376144 11424 consensus_queue.cc:260] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.398433 11553 consensus_queue.cc:1048] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260321 02:58:21.401335 11424 ts_tablet_manager.cc:1434] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.042s user 0.005s sys 0.002s
I20260321 02:58:21.395864 11531 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:21.402175 11531 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.405125 11531 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:21.406843 11424 tablet_bootstrap.cc:492] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:21.410511 11429 tablet_bootstrap.cc:492] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:21.411111 11429 ts_tablet_manager.cc:1403] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.014s
I20260321 02:58:21.415199 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:21.416477 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:21.417140 11385 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
W20260321 02:58:21.419404 11200 leader_election.cc:343] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:21.420272 11429 raft_consensus.cc:359] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.420975 11429 raft_consensus.cc:385] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:21.421267 11429 raft_consensus.cc:740] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:21.421213 11531 consensus_queue.cc:1048] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:21.422317 11200 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:21.422050 11429 consensus_queue.cc:260] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.430152 11459 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:21.430678 11459 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.431107 11429 ts_tablet_manager.cc:1434] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.020s user 0.009s sys 0.000s
I20260321 02:58:21.432564 11429 tablet_bootstrap.cc:492] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:21.428246 11446 raft_consensus.cc:2804] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:21.433331 11459 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:21.433534 11446 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:21.433935 11446 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:21.439226 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:21.440002 11309 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:21.441926 11349 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:21.443106 11459 raft_consensus.cc:2804] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:21.443552 11459 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:21.444166 11459 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:21.450570 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:21.459183 11424 tablet_bootstrap.cc:654] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:21.513332 11446 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.517545 11425 tablet_bootstrap.cc:492] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:21.529381 11425 ts_tablet_manager.cc:1403] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.186s user 0.012s sys 0.015s
I20260321 02:58:21.518783 11428 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:21.532343 11428 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.534184 11428 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:21.534658 11428 consensus_queue.cc:1048] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:21.521548 11311 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:21.538292 11425 raft_consensus.cc:359] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.539021 11425 raft_consensus.cc:385] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:21.521523 11429 tablet_bootstrap.cc:654] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:21.527523 11531 consensus_queue.cc:1048] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:21.539286 11425 raft_consensus.cc:740] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:21.457502 11236 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:21.539983 11425 consensus_queue.cc:260] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.542514 11425 ts_tablet_manager.cc:1434] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260321 02:58:21.543632 11425 tablet_bootstrap.cc:492] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:21.548821 11459 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.550992 11459 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:21.559405 11446 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:21.571280 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:21.572204 11383 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:21.571437 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:21.575004 11309 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:21.575559 11424 tablet_bootstrap.cc:492] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:21.576175 11424 ts_tablet_manager.cc:1403] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.170s user 0.011s sys 0.010s
I20260321 02:58:21.575023 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:21.577584 11307 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:21.577482 11459 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:21.583916 11459 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.586244 11429 tablet_bootstrap.cc:492] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:21.586839 11429 ts_tablet_manager.cc:1403] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.154s user 0.010s sys 0.003s
I20260321 02:58:21.577289 11553 consensus_queue.cc:1048] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260321 02:58:21.600097 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:21.589140 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:21.654639 11429 raft_consensus.cc:359] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.655679 11429 raft_consensus.cc:385] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:21.656046 11429 raft_consensus.cc:740] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:21.665977 11425 tablet_bootstrap.cc:654] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:21.647303 11487 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:21.646762 11540 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:21.685928 11429 consensus_queue.cc:260] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.687177 11428 consensus_queue.cc:1048] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:21.685586 11424 raft_consensus.cc:359] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.689107 11424 raft_consensus.cc:385] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:21.689529 11424 raft_consensus.cc:740] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
W20260321 02:58:21.666646 11182 auto_rebalancer.cc:247] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260321 02:58:21.690547 11446 consensus_queue.cc:1048] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260321 02:58:21.694955 11459 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:21.689577 11540 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.690033 11424 consensus_queue.cc:260] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.684003 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:21.702243 11429 ts_tablet_manager.cc:1434] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.115s user 0.014s sys 0.000s
I20260321 02:58:21.710785 11307 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:21.738373 11424 ts_tablet_manager.cc:1434] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.162s user 0.026s sys 0.060s
I20260321 02:58:21.739373 11424 tablet_bootstrap.cc:492] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:21.742689 11429 tablet_bootstrap.cc:492] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:21.689640 11487 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.741660 11233 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:21.702626 11383 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:21.708833 11531 consensus_queue.cc:1048] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.057s
I20260321 02:58:21.767390 11537 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:21.761929 11309 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:21.775767 11492 consensus_queue.cc:1048] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260321 02:58:21.785280 11537 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.808173 11200 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:21.825434 11540 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:21.836683 11425 tablet_bootstrap.cc:492] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:21.837205 11425 ts_tablet_manager.cc:1403] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.294s user 0.013s sys 0.012s
I20260321 02:58:21.830313 11616 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:21.831426 11446 raft_consensus.cc:2804] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:21.834201 11626 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:21.844726 11616 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.846711 11616 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:58:21.844398 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 198965us
W20260321 02:58:21.848088 11349 leader_election.cc:343] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:21.848760 11349 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:21.849553 11582 raft_consensus.cc:2804] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:21.849454 11233 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:21.850052 11582 raft_consensus.cc:697] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:21.851722 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:21.852048 11537 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:21.839226 11487 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:21.844229 11626 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.854197 11424 tablet_bootstrap.cc:654] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:21.854374 11446 raft_consensus.cc:697] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:21.861680 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:21.865159 11446 consensus_queue.cc:237] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.866190 11425 raft_consensus.cc:359] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.866928 11425 raft_consensus.cc:385] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:21.867329 11425 raft_consensus.cc:740] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:21.868346 11425 consensus_queue.cc:260] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.869963 11425 ts_tablet_manager.cc:1434] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.032s user 0.005s sys 0.001s
I20260321 02:58:21.870626 11429 tablet_bootstrap.cc:654] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:21.870836 11425 tablet_bootstrap.cc:492] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:21.873510 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:21.875887 11582 consensus_queue.cc:237] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:21.871269 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:21.927999 11626 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:21.931850 11233 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
I20260321 02:58:21.943624 11275 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:21.945158 11626 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:21.949267 11626 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:21.914232 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 50656.6us
I20260321 02:58:21.964977 11626 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:21.963099 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:21.973083 11537 raft_consensus.cc:2804] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260321 02:58:21.981925 11349 leader_election.cc:343] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:21.984215 11349 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:21.985360 11428 raft_consensus.cc:2749] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:21.965341 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:21.986256 11537 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:21.992895 11383 raft_consensus.cc:2468] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:22.008622 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:22.009346 11380 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
W20260321 02:58:22.009439 11200 leader_election.cc:343] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:22.006285 11537 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
W20260321 02:58:22.012835 11349 leader_election.cc:343] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:22.015805 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:22.016119 11549 consensus_queue.cc:1048] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:22.018612 11429 tablet_bootstrap.cc:492] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:22.019024 11429 ts_tablet_manager.cc:1403] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.277s user 0.034s sys 0.015s
I20260321 02:58:22.021497 11429 raft_consensus.cc:359] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.022145 11429 raft_consensus.cc:385] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.022418 11429 raft_consensus.cc:740] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.022946 11429 consensus_queue.cc:260] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.033339 11425 tablet_bootstrap.cc:654] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.035243 11429 ts_tablet_manager.cc:1434] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260321 02:58:22.036260 11429 tablet_bootstrap.cc:492] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:22.040568 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:22.049888 11130 catalog_manager.cc:5671] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:22.050889 11549 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:22.051465 11549 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.062702 11549 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:22.067541 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:22.068224 11384 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:22.073864 11132 catalog_manager.cc:5671] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:22.079648 11429 tablet_bootstrap.cc:654] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.075475 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:22.094345 11424 tablet_bootstrap.cc:492] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:22.094977 11424 ts_tablet_manager.cc:1403] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.356s user 0.026s sys 0.029s
I20260321 02:58:22.096182 11233 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:22.103796 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:22.099756 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:22.112449 11200 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:22.111569 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:22.116895 11349 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:22.119642 11428 raft_consensus.cc:2804] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:22.120049 11428 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:22.120321 11428 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.120482 11429 tablet_bootstrap.cc:492] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:22.120728 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:22.119339 11200 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:22.120972 11429 ts_tablet_manager.cc:1403] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.085s user 0.013s sys 0.009s
W20260321 02:58:22.122011 11200 leader_election.cc:343] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:22.129171 11446 raft_consensus.cc:2804] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:22.129652 11446 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:22.129374 11424 raft_consensus.cc:359] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.129997 11446 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.130251 11537 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.130126 11424 raft_consensus.cc:385] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.116173 11626 raft_consensus.cc:2804] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:22.132472 11626 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:22.132195 11424 raft_consensus.cc:740] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.132750 11537 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:22.132853 11626 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.133797 11424 consensus_queue.cc:260] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.134677 11311 raft_consensus.cc:2393] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 1: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:22.121812 11384 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
W20260321 02:58:22.155536 11349 leader_election.cc:343] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:22.118410 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:22.181411 11200 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:58:22.183547 11349 leader_election.cc:343] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:22.184146 11349 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:22.185633 11582 raft_consensus.cc:2749] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:22.185681 11630 raft_consensus.cc:2804] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:22.186290 11630 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:22.187214 11630 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.196281 11616 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:58:22.198053 11349 leader_election.cc:343] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:22.197274 11616 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:22.200166 11639 log.cc:927] Time spent T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.092s user 0.001s sys 0.000s
I20260321 02:58:22.201654 11616 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:22.202216 11626 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.199450 11428 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.208216 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:22.209944 11424 ts_tablet_manager.cc:1434] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.115s user 0.009s sys 0.000s
I20260321 02:58:22.211089 11424 tablet_bootstrap.cc:492] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:22.213835 11429 raft_consensus.cc:359] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.214301 11446 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.215409 11428 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:22.216087 11429 raft_consensus.cc:385] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.216429 11429 raft_consensus.cc:740] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.219237 11425 tablet_bootstrap.cc:492] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:22.217100 11429 consensus_queue.cc:260] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.219844 11425 ts_tablet_manager.cc:1403] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.349s user 0.032s sys 0.039s
I20260321 02:58:22.215330 11626 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:22.223069 11446 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:22.223255 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:58:22.220559 11349 leader_election.cc:343] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:22.227315 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:22.228204 11232 raft_consensus.cc:2393] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 1: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:22.230674 11384 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.230890 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:58:22.232268 11200 leader_election.cc:343] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:22.233942 11429 ts_tablet_manager.cc:1434] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.112s user 0.012s sys 0.005s
I20260321 02:58:22.234723 11429 tablet_bootstrap.cc:492] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:22.233393 11275 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:22.238119 11620 raft_consensus.cc:2749] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:22.239251 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:22.239869 11308 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
W20260321 02:58:22.246665 11349 leader_election.cc:343] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:22.247102 11349 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:22.247686 11428 raft_consensus.cc:2749] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:22.253185 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:22.253572 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:22.253822 11385 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.256532 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:22.256613 11310 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:22.260702 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:22.251699 11630 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.261929 11308 raft_consensus.cc:2468] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:22.264510 11235 raft_consensus.cc:2393] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 1: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:22.265566 11630 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:22.266242 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:58:22.268589 11349 leader_election.cc:343] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:22.276969 11310 raft_consensus.cc:2393] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 1: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:22.277616 11349 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:22.277962 11200 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:22.278870 11630 raft_consensus.cc:2804] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:22.278764 11582 raft_consensus.cc:2749] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260321 02:58:22.280090 11630 raft_consensus.cc:697] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:22.281029 11630 consensus_queue.cc:237] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.285130 11349 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:22.286154 11582 raft_consensus.cc:2804] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:22.286540 11582 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:22.286893 11582 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.271320 11385 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:22.271819 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:22.272667 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:22.274760 11425 raft_consensus.cc:359] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.296155 11425 raft_consensus.cc:385] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.296605 11425 raft_consensus.cc:740] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.275691 11384 raft_consensus.cc:2468] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:22.299250 11383 raft_consensus.cc:2393] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 1: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:58:22.301714 11379 raft_consensus.cc:2393] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 1: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:22.302416 11275 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:22.302695 11200 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:22.303757 11424 tablet_bootstrap.cc:654] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.297870 11425 consensus_queue.cc:260] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.306252 11537 raft_consensus.cc:2804] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:22.306733 11537 raft_consensus.cc:697] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:58:22.305390 11446 raft_consensus.cc:2749] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260321 02:58:22.305897 11200 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:22.313011 11425 ts_tablet_manager.cc:1434] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.092s user 0.004s sys 0.003s
I20260321 02:58:22.313822 11425 tablet_bootstrap.cc:492] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:22.307860 11537 consensus_queue.cc:237] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.321169 11429 tablet_bootstrap.cc:654] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.323895 11637 raft_consensus.cc:2749] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260321 02:58:22.345042 11425 tablet_bootstrap.cc:654] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.340528 11582 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.350692 11429 tablet_bootstrap.cc:492] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:22.351238 11429 ts_tablet_manager.cc:1403] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.117s user 0.011s sys 0.006s
I20260321 02:58:22.351454 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:58:22.352885 11349 leader_election.cc:343] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:22.354065 11582 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:22.348294 11424 tablet_bootstrap.cc:492] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:22.354926 11424 ts_tablet_manager.cc:1403] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.144s user 0.018s sys 0.000s
I20260321 02:58:22.354562 11429 raft_consensus.cc:359] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.355480 11429 raft_consensus.cc:385] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.355822 11429 raft_consensus.cc:740] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.356549 11429 consensus_queue.cc:260] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.358297 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:22.358078 11424 raft_consensus.cc:359] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.358901 11424 raft_consensus.cc:385] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.358915 11310 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.359196 11424 raft_consensus.cc:740] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.359201 11429 ts_tablet_manager.cc:1434] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260321 02:58:22.360231 11424 consensus_queue.cc:260] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.360918 11429 tablet_bootstrap.cc:492] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:22.362923 11424 ts_tablet_manager.cc:1434] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260321 02:58:22.366467 11429 tablet_bootstrap.cc:654] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.366968 11630 consensus_queue.cc:1048] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:22.374449 11637 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:22.375264 11637 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.378089 11637 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:22.382669 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:22.383680 11308 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:22.386587 11200 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:22.387403 11630 raft_consensus.cc:2804] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:22.387869 11630 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:22.388347 11630 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.389729 11132 catalog_manager.cc:5671] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:22.391644 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:22.396121 11385 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:22.395332 11424 tablet_bootstrap.cc:492] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:22.401633 11310 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:22.405218 11349 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:22.411754 11622 raft_consensus.cc:2804] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:22.414928 11622 raft_consensus.cc:697] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:22.420553 11622 consensus_queue.cc:237] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.427088 11537 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:22.427640 11537 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.433967 11537 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:22.436379 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:22.437081 11385 raft_consensus.cc:2393] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 1: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:58:22.438226 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:22.439060 11232 raft_consensus.cc:2393] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 1: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:22.440618 11275 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:22.441604 11537 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.411342 11630 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.445480 11429 tablet_bootstrap.cc:492] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:22.446203 11133 catalog_manager.cc:5671] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:22.451679 11429 ts_tablet_manager.cc:1403] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.091s user 0.031s sys 0.044s
I20260321 02:58:22.455490 11429 raft_consensus.cc:359] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.456269 11429 raft_consensus.cc:385] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.456542 11429 raft_consensus.cc:740] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.449553 11425 tablet_bootstrap.cc:492] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:22.458359 11425 ts_tablet_manager.cc:1403] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.145s user 0.031s sys 0.038s
I20260321 02:58:22.460687 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:22.461301 11308 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.460923 11429 consensus_queue.cc:260] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.463533 11425 raft_consensus.cc:359] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.464269 11425 raft_consensus.cc:385] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.464561 11429 ts_tablet_manager.cc:1434] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260321 02:58:22.464538 11425 raft_consensus.cc:740] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.465745 11429 tablet_bootstrap.cc:492] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:22.465413 11425 consensus_queue.cc:260] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.468050 11425 ts_tablet_manager.cc:1434] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260321 02:58:22.468971 11425 tablet_bootstrap.cc:492] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:22.470623 11582 consensus_queue.cc:1048] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:22.474128 11308 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:22.449717 11537 raft_consensus.cc:2749] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:22.476681 11630 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:22.477144 11634 consensus_queue.cc:1048] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:22.479055 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:22.480197 11429 tablet_bootstrap.cc:654] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.484257 11385 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.475903 11425 tablet_bootstrap.cc:654] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.480317 11424 tablet_bootstrap.cc:654] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.494603 11429 tablet_bootstrap.cc:492] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:22.495102 11429 ts_tablet_manager.cc:1403] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.010s
I20260321 02:58:22.497742 11429 raft_consensus.cc:359] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.498464 11429 raft_consensus.cc:385] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.498742 11429 raft_consensus.cc:740] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.502946 11134 catalog_manager.cc:5671] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:22.506318 11385 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:22.511904 11200 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:22.513331 11634 raft_consensus.cc:2804] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:22.513154 11429 consensus_queue.cc:260] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.514098 11634 raft_consensus.cc:697] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:22.515275 11634 consensus_queue.cc:237] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.520418 11429 ts_tablet_manager.cc:1434] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.025s user 0.008s sys 0.001s
I20260321 02:58:22.521423 11429 tablet_bootstrap.cc:492] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:22.534497 11424 tablet_bootstrap.cc:492] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:22.535136 11424 ts_tablet_manager.cc:1403] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.143s user 0.013s sys 0.000s
I20260321 02:58:22.536080 11429 tablet_bootstrap.cc:654] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.544641 11446 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:22.545190 11446 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.547155 11446 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:22.553848 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:22.554574 11307 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:22.557317 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:22.561556 11383 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:22.563833 11425 tablet_bootstrap.cc:492] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:22.564452 11425 ts_tablet_manager.cc:1403] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.096s user 0.016s sys 0.030s
I20260321 02:58:22.566304 11200 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:22.568480 11429 tablet_bootstrap.cc:492] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:22.569278 11429 ts_tablet_manager.cc:1403] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.005s
I20260321 02:58:22.580936 11424 raft_consensus.cc:359] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.581667 11424 raft_consensus.cc:385] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.576025 11446 raft_consensus.cc:2804] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:22.581948 11424 raft_consensus.cc:740] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.582257 11446 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:22.582616 11446 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.582859 11424 consensus_queue.cc:260] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.585156 11424 ts_tablet_manager.cc:1434] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.050s user 0.006s sys 0.003s
I20260321 02:58:22.594038 11425 raft_consensus.cc:359] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.594772 11425 raft_consensus.cc:385] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.595039 11425 raft_consensus.cc:740] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.595649 11425 consensus_queue.cc:260] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.597262 11425 ts_tablet_manager.cc:1434] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20260321 02:58:22.598160 11425 tablet_bootstrap.cc:492] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:22.590467 11424 tablet_bootstrap.cc:492] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:22.600957 11622 consensus_queue.cc:1048] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:22.602659 11134 catalog_manager.cc:5671] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:22.603467 11446 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.609040 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:22.609886 11383 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.609758 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:22.610818 11307 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.605564 11446 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:22.625066 11383 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:22.631278 11307 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:22.634094 11200 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:22.637383 11634 raft_consensus.cc:2804] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:22.638047 11634 raft_consensus.cc:697] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:22.639132 11634 consensus_queue.cc:237] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.648591 11429 raft_consensus.cc:359] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.649632 11429 raft_consensus.cc:385] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.650414 11429 raft_consensus.cc:740] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.651609 11429 consensus_queue.cc:260] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.654502 11429 ts_tablet_manager.cc:1434] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.085s user 0.012s sys 0.000s
I20260321 02:58:22.655426 11429 tablet_bootstrap.cc:492] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:22.657720 11134 catalog_manager.cc:5671] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:22.660989 11424 tablet_bootstrap.cc:654] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.665993 11425 tablet_bootstrap.cc:654] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.673168 11429 tablet_bootstrap.cc:654] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.673662 11425 tablet_bootstrap.cc:492] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:22.674405 11425 ts_tablet_manager.cc:1403] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.008s
I20260321 02:58:22.677439 11425 raft_consensus.cc:359] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.677992 11425 raft_consensus.cc:385] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.678220 11425 raft_consensus.cc:740] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.678987 11425 consensus_queue.cc:260] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.680855 11425 ts_tablet_manager.cc:1434] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260321 02:58:22.681612 11425 tablet_bootstrap.cc:492] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:22.689041 11425 tablet_bootstrap.cc:654] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.692637 11424 tablet_bootstrap.cc:492] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:22.693104 11424 ts_tablet_manager.cc:1403] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.103s user 0.010s sys 0.016s
I20260321 02:58:22.701565 11429 tablet_bootstrap.cc:492] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:22.702934 11429 ts_tablet_manager.cc:1403] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.048s user 0.014s sys 0.012s
I20260321 02:58:22.704061 11425 tablet_bootstrap.cc:492] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:22.704470 11425 ts_tablet_manager.cc:1403] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.010s
I20260321 02:58:22.705508 11429 raft_consensus.cc:359] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.706241 11429 raft_consensus.cc:385] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.706511 11429 raft_consensus.cc:740] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.707157 11429 consensus_queue.cc:260] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.707732 11425 raft_consensus.cc:359] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.708482 11425 raft_consensus.cc:385] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.708740 11425 raft_consensus.cc:740] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.709345 11425 consensus_queue.cc:260] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.697007 11424 raft_consensus.cc:359] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.710232 11424 raft_consensus.cc:385] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.710515 11424 raft_consensus.cc:740] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.711131 11424 consensus_queue.cc:260] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.711618 11425 ts_tablet_manager.cc:1434] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260321 02:58:22.713361 11424 ts_tablet_manager.cc:1434] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.020s user 0.002s sys 0.004s
I20260321 02:58:22.714300 11429 ts_tablet_manager.cc:1434] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260321 02:58:22.715123 11627 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:22.715665 11627 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.716234 11424 tablet_bootstrap.cc:492] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:22.717725 11627 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:22.719841 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:22.715260 11429 tablet_bootstrap.cc:492] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
W20260321 02:58:22.721627 11275 leader_election.cc:343] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:22.723883 11425 tablet_bootstrap.cc:492] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:22.724118 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:58:22.730552 11182 auto_rebalancer.cc:247] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260321 02:58:22.731413 11424 tablet_bootstrap.cc:654] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
W20260321 02:58:22.731570 11275 leader_election.cc:343] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:22.732004 11275 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:22.733362 11537 raft_consensus.cc:2749] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:22.735394 11425 tablet_bootstrap.cc:654] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.739293 11429 tablet_bootstrap.cc:654] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.743507 11630 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:22.744246 11630 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.746248 11630 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:22.749455 11424 tablet_bootstrap.cc:492] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:22.750346 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:22.750694 11424 ts_tablet_manager.cc:1403] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.002s
I20260321 02:58:22.753201 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:22.749809 11425 tablet_bootstrap.cc:492] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:22.750114 11429 tablet_bootstrap.cc:492] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:22.753885 11383 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:22.754103 11429 ts_tablet_manager.cc:1403] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.000s
I20260321 02:58:22.754500 11424 raft_consensus.cc:359] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.755084 11200 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:22.755942 11425 ts_tablet_manager.cc:1403] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.004s
W20260321 02:58:22.756641 11200 leader_election.cc:343] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:22.756839 11630 raft_consensus.cc:2804] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:22.757169 11630 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:22.757494 11630 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.756666 11429 raft_consensus.cc:359] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.758385 11429 raft_consensus.cc:385] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.758632 11429 raft_consensus.cc:740] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.759209 11425 raft_consensus.cc:359] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.759382 11429 consensus_queue.cc:260] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.759915 11425 raft_consensus.cc:385] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.760249 11425 raft_consensus.cc:740] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.760871 11425 consensus_queue.cc:260] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.762981 11425 ts_tablet_manager.cc:1434] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260321 02:58:22.763375 11616 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:22.763862 11616 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.765763 11616 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:22.766834 11429 ts_tablet_manager.cc:1434] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260321 02:58:22.755273 11424 raft_consensus.cc:385] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.767807 11429 tablet_bootstrap.cc:492] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:22.767848 11424 raft_consensus.cc:740] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.768055 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:22.769004 11424 consensus_queue.cc:260] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:22.769747 11349 leader_election.cc:343] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:22.771621 11630 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.772881 11634 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:22.773365 11634 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.774784 11429 tablet_bootstrap.cc:654] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.776314 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:22.776747 11425 tablet_bootstrap.cc:492] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:22.776873 11307 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.777828 11630 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:22.775889 11424 ts_tablet_manager.cc:1434] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260321 02:58:22.779048 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:22.778977 11537 consensus_queue.cc:1048] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:22.784070 11425 tablet_bootstrap.cc:654] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.785142 11634 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:22.786937 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:22.788968 11308 raft_consensus.cc:2393] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 1: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
W20260321 02:58:22.790680 11349 leader_election.cc:343] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:22.791100 11349 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:22.792505 11616 raft_consensus.cc:2749] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:22.786077 11307 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:22.794821 11200 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:22.795599 11634 raft_consensus.cc:2804] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:22.797173 11634 raft_consensus.cc:697] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:22.798199 11634 consensus_queue.cc:237] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.801532 11425 tablet_bootstrap.cc:492] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:22.802016 11425 ts_tablet_manager.cc:1403] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.013s
I20260321 02:58:22.804605 11429 tablet_bootstrap.cc:492] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:22.805305 11429 ts_tablet_manager.cc:1403] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.038s user 0.015s sys 0.001s
I20260321 02:58:22.806712 11425 raft_consensus.cc:359] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.807502 11425 raft_consensus.cc:385] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.807861 11425 raft_consensus.cc:740] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.808017 11429 raft_consensus.cc:359] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.809082 11429 raft_consensus.cc:385] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.809238 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:22.809432 11429 raft_consensus.cc:740] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.809545 11425 consensus_queue.cc:260] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.810504 11429 consensus_queue.cc:260] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.812795 11425 ts_tablet_manager.cc:1434] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20260321 02:58:22.812795 11429 ts_tablet_manager.cc:1434] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260321 02:58:22.812822 11627 consensus_queue.cc:1048] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:22.814218 11425 tablet_bootstrap.cc:492] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:22.809101 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:22.815377 11383 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.815519 11200 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:22.816947 11446 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.821091 11425 tablet_bootstrap.cc:654] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.821906 11429 tablet_bootstrap.cc:492] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:22.822518 11383 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:22.824887 11627 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:22.825392 11627 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.825881 11446 raft_consensus.cc:2749] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:22.826843 11622 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:22.827479 11622 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.830504 11622 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:22.832346 11424 tablet_bootstrap.cc:492] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:22.833323 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:22.836517 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:22.837082 11627 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:22.837234 11232 raft_consensus.cc:2468] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
I20260321 02:58:22.838395 11275 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:58:22.840939 11349 leader_election.cc:343] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:22.841214 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:22.842139 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:22.842139 11383 raft_consensus.cc:2468] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
W20260321 02:58:22.844659 11349 leader_election.cc:343] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:22.843887 11627 raft_consensus.cc:2804] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:22.845075 11349 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:22.845305 11627 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:22.845779 11549 raft_consensus.cc:2749] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:22.846426 11627 raft_consensus.cc:3060] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.840986 11622 consensus_queue.cc:1048] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260321 02:58:22.855005 11424 tablet_bootstrap.cc:654] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.876962 11627 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.879158 11627 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:22.881958 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:22.882983 11385 raft_consensus.cc:3060] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.894479 11385 raft_consensus.cc:2468] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:22.898238 11275 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:22.898559 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:22.917644 11429 tablet_bootstrap.cc:654] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:22.915815 11130 catalog_manager.cc:5671] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:22.928184 11631 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:22.928236 11235 raft_consensus.cc:3060] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.928789 11631 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.931319 11631 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:22.933609 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:58:22.937413 11200 leader_election.cc:343] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:22.950191 11425 tablet_bootstrap.cc:492] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:22.950784 11425 ts_tablet_manager.cc:1403] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.137s user 0.052s sys 0.034s
I20260321 02:58:22.955091 11425 raft_consensus.cc:359] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.958222 11622 consensus_queue.cc:1048] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:22.951473 11681 raft_consensus.cc:2804] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:22.960351 11425 raft_consensus.cc:385] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:22.959566 11235 raft_consensus.cc:2468] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:22.961252 11425 raft_consensus.cc:740] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:22.961280 11616 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:22.963249 11616 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.965231 11616 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:22.967125 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:22.967850 11308 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:22.969935 11349 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:22.970652 11622 raft_consensus.cc:2804] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:22.971019 11622 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:22.971336 11622 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:22.967855 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:58:22.977442 11200 leader_election.cc:343] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:22.978052 11200 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:22.981014 11424 tablet_bootstrap.cc:492] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:22.982115 11424 ts_tablet_manager.cc:1403] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.150s user 0.042s sys 0.019s
I20260321 02:58:22.983388 11634 raft_consensus.cc:2749] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:22.962128 11425 consensus_queue.cc:260] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:22.989871 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:22.990571 11235 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:22.991982 11446 consensus_queue.cc:1048] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:23.011590 11425 ts_tablet_manager.cc:1434] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.061s user 0.005s sys 0.005s
I20260321 02:58:23.012950 11681 raft_consensus.cc:697] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:58:23.014318 11681 consensus_queue.cc:237] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.042455 11425 tablet_bootstrap.cc:492] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:23.044184 11424 raft_consensus.cc:359] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.045396 11424 raft_consensus.cc:385] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:23.045853 11424 raft_consensus.cc:740] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:23.046823 11424 consensus_queue.cc:260] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.050313 11429 tablet_bootstrap.cc:492] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:23.050954 11429 ts_tablet_manager.cc:1403] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.229s user 0.018s sys 0.033s
I20260321 02:58:23.120983 11424 ts_tablet_manager.cc:1434] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.138s user 0.008s sys 0.010s
I20260321 02:58:23.136276 11424 tablet_bootstrap.cc:492] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:23.119745 11688 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:58:23.223608 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 81080.9us
I20260321 02:58:23.155705 11688 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.231101 11429 raft_consensus.cc:359] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.231720 11429 raft_consensus.cc:385] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:23.231997 11429 raft_consensus.cc:740] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
W20260321 02:58:23.232196 11678 log.cc:927] Time spent T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.075s user 0.005s sys 0.008s
I20260321 02:58:23.237682 11622 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.239519 11688 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:23.239702 11622 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:23.240365 11429 consensus_queue.cc:260] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.265576 11429 ts_tablet_manager.cc:1434] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.214s user 0.006s sys 0.004s
I20260321 02:58:23.266785 11429 tablet_bootstrap.cc:492] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
W20260321 02:58:23.270103 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 127836us
I20260321 02:58:23.278913 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:23.279390 11311 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:23.293073 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:23.294931 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:23.295652 11232 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:23.296034 11631 consensus_queue.cc:1048] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:23.293793 11383 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
I20260321 02:58:23.308907 11311 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
W20260321 02:58:23.343511 11692 log.cc:927] Time spent T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
I20260321 02:58:23.315819 11349 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:23.354857 11428 raft_consensus.cc:2804] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:23.355486 11428 raft_consensus.cc:697] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:23.356456 11428 consensus_queue.cc:237] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.370854 11275 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:58:23.373107 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 46670.8us
I20260321 02:58:23.378364 11698 consensus_queue.cc:1048] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20260321 02:58:23.381999 11696 log.cc:927] Time spent T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
I20260321 02:58:23.372469 11634 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:23.389984 11424 tablet_bootstrap.cc:654] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:23.375573 11691 raft_consensus.cc:2804] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:23.369875 11703 consensus_queue.cc:1048] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20260321 02:58:23.379557 11637 consensus_queue.cc:1048] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260321 02:58:23.393026 11634 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.411538 11425 tablet_bootstrap.cc:654] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:23.397172 11691 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:23.414525 11691 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:23.428174 11429 tablet_bootstrap.cc:654] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:23.414021 11634 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:23.419510 11699 consensus_queue.cc:1048] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260321 02:58:23.447292 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:58:23.449410 11275 leader_election.cc:343] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:23.452822 11134 catalog_manager.cc:5671] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:23.464360 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:58:23.495941 11200 leader_election.cc:343] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:23.455863 11691 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.523326 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:23.537716 11424 tablet_bootstrap.cc:492] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:23.538434 11424 ts_tablet_manager.cc:1403] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.413s user 0.017s sys 0.013s
I20260321 02:58:23.540691 11691 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:23.543366 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:23.544044 11379 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:23.543305 11232 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:23.549252 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:23.552634 11705 consensus_queue.cc:1048] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
W20260321 02:58:23.555459 11275 leader_election.cc:343] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:23.559141 11379 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:23.564304 11275 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:23.565255 11688 raft_consensus.cc:2804] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:23.565830 11688 raft_consensus.cc:697] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:58:23.566944 11688 consensus_queue.cc:237] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:23.570264 11200 leader_election.cc:343] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:23.570670 11699 consensus_queue.cc:1048] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:23.570722 11200 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:23.573700 11424 raft_consensus.cc:359] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.574376 11424 raft_consensus.cc:385] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:23.574642 11424 raft_consensus.cc:740] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:23.575492 11424 consensus_queue.cc:260] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.577669 11705 raft_consensus.cc:2749] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:23.600965 11424 ts_tablet_manager.cc:1434] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.062s user 0.009s sys 0.000s
I20260321 02:58:23.602299 11424 tablet_bootstrap.cc:492] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:23.617560 11689 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:23.620473 11425 tablet_bootstrap.cc:492] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:23.620972 11425 ts_tablet_manager.cc:1403] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.579s user 0.041s sys 0.026s
I20260321 02:58:23.625238 11425 raft_consensus.cc:359] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.625897 11425 raft_consensus.cc:385] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:23.626147 11425 raft_consensus.cc:740] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:23.626714 11425 consensus_queue.cc:260] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.630774 11425 ts_tablet_manager.cc:1434] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260321 02:58:23.648633 11429 tablet_bootstrap.cc:492] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:23.649292 11429 ts_tablet_manager.cc:1403] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.383s user 0.015s sys 0.026s
I20260321 02:58:23.618116 11689 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.660925 11695 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:23.650120 11691 consensus_queue.cc:1048] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260321 02:58:23.670323 11689 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:23.671885 11695 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.680814 11695 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:23.681217 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:23.682277 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:23.686419 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:58:23.702608 11275 leader_election.cc:343] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:23.694319 11383 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
I20260321 02:58:23.694629 11425 tablet_bootstrap.cc:492] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:23.694562 11424 tablet_bootstrap.cc:654] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:23.735569 11429 raft_consensus.cc:359] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.741636 11429 raft_consensus.cc:385] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:23.741974 11429 raft_consensus.cc:740] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:23.742726 11429 consensus_queue.cc:260] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.744344 11429 ts_tablet_manager.cc:1434] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.095s user 0.010s sys 0.000s
I20260321 02:58:23.747931 11429 tablet_bootstrap.cc:492] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:23.692453 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:58:23.762202 11349 leader_election.cc:343] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
W20260321 02:58:23.766119 11349 leader_election.cc:343] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:23.766688 11349 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:23.769495 11275 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:23.784353 11686 raft_consensus.cc:2804] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:23.784847 11686 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:23.795271 11686 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:23.841949 11629 raft_consensus.cc:2749] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260321 02:58:23.844571 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 78494.5us
W20260321 02:58:23.880915 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 111519us
I20260321 02:58:23.895701 11130 catalog_manager.cc:5671] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:23.890935 11694 consensus_queue.cc:1048] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260321 02:58:23.921566 11428 consensus_queue.cc:1048] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:23.933044 11429 tablet_bootstrap.cc:654] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:23.935314 11549 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:23.935848 11549 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.937780 11549 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:23.940706 11686 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.942638 11686 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:23.942623 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:23.943728 11235 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:23.946058 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:23.947969 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:23.948696 11308 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
W20260321 02:58:23.950042 11275 leader_election.cc:343] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:23.954902 11349 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:23.956234 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:23.956836 11379 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:23.957882 11695 raft_consensus.cc:2804] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:23.958298 11695 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:23.958647 11695 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:23.977308 11424 tablet_bootstrap.cc:492] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:23.977895 11424 ts_tablet_manager.cc:1403] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.376s user 0.026s sys 0.041s
I20260321 02:58:23.970508 11537 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:23.982859 11537 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:23.984961 11537 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:23.988857 11379 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:24.001457 11275 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:23.995939 11134 catalog_manager.cc:5671] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:24.009294 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:24.007308 11537 consensus_queue.cc:1048] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260321 02:58:24.008543 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:58:24.011284 11182 auto_rebalancer.cc:247] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260321 02:58:24.011123 11275 leader_election.cc:343] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:24.013288 11429 tablet_bootstrap.cc:492] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:24.014175 11429 ts_tablet_manager.cc:1403] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.266s user 0.020s sys 0.022s
W20260321 02:58:24.015350 11275 leader_election.cc:343] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:24.015743 11275 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:24.020982 11429 raft_consensus.cc:359] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.021790 11429 raft_consensus.cc:385] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.007086 11684 raft_consensus.cc:2804] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:24.022071 11429 raft_consensus.cc:740] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.022272 11684 raft_consensus.cc:697] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:58:24.022691 11429 consensus_queue.cc:260] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.023065 11684 consensus_queue.cc:237] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.025393 11429 ts_tablet_manager.cc:1434] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260321 02:58:24.026226 11429 tablet_bootstrap.cc:492] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:24.036365 11425 tablet_bootstrap.cc:654] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.017050 11537 raft_consensus.cc:2749] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:24.030973 11691 consensus_queue.cc:1048] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260321 02:58:24.042546 11702 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:24.056130 11702 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.058295 11702 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:24.064524 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:58:24.091110 11200 leader_election.cc:343] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:24.094208 11424 raft_consensus.cc:359] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.099083 11424 raft_consensus.cc:385] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.099515 11424 raft_consensus.cc:740] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.100783 11424 consensus_queue.cc:260] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.106997 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:24.107592 11379 raft_consensus.cc:2468] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:24.097837 11695 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.111159 11424 ts_tablet_manager.cc:1434] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.133s user 0.009s sys 0.001s
I20260321 02:58:24.112366 11424 tablet_bootstrap.cc:492] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:24.117341 11200 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:58:24.121232 11731 log.cc:927] Time spent T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260321 02:58:24.122598 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:24.122984 11429 tablet_bootstrap.cc:654] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.123237 11307 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:24.111171 11695 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:24.121997 11702 raft_consensus.cc:2804] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:24.129222 11702 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:24.127408 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:24.134325 11235 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:24.132964 11702 raft_consensus.cc:3060] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.152460 11307 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
W20260321 02:58:24.154772 11275 consensus_peers.cc:597] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da -> Peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Couldn't send request to peer 0d7efc590a1041a9bd4cd4a971332dfb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:24.155565 11134 catalog_manager.cc:5671] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:24.160151 11424 tablet_bootstrap.cc:654] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.166239 11425 tablet_bootstrap.cc:492] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:24.167892 11425 ts_tablet_manager.cc:1403] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.474s user 0.022s sys 0.033s
I20260321 02:58:24.167507 11702 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.171777 11235 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:24.172784 11349 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:24.174084 11695 raft_consensus.cc:2804] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:58:24.174803 11695 raft_consensus.cc:697] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 2 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:24.175613 11695 consensus_queue.cc:237] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.176945 11702 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:24.182956 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:24.183591 11379 raft_consensus.cc:3060] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.177430 11425 raft_consensus.cc:359] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.187458 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:24.188541 11425 raft_consensus.cc:385] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.189111 11425 raft_consensus.cc:740] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.180584 11429 tablet_bootstrap.cc:492] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:24.179423 11424 tablet_bootstrap.cc:492] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:24.192654 11429 ts_tablet_manager.cc:1403] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.167s user 0.010s sys 0.005s
I20260321 02:58:24.196376 11429 raft_consensus.cc:359] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.198318 11429 raft_consensus.cc:385] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.198683 11429 raft_consensus.cc:740] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.199332 11429 consensus_queue.cc:260] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.192719 11424 ts_tablet_manager.cc:1403] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.081s user 0.007s sys 0.004s
I20260321 02:58:24.191896 11379 raft_consensus.cc:2468] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
W20260321 02:58:24.197338 11200 leader_election.cc:343] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:24.191697 11425 consensus_queue.cc:260] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.202837 11200 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:24.203830 11702 raft_consensus.cc:2804] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:24.204291 11424 raft_consensus.cc:359] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.204336 11702 raft_consensus.cc:697] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:24.205138 11424 raft_consensus.cc:385] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.205436 11424 raft_consensus.cc:740] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.205796 11702 consensus_queue.cc:237] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.206036 11424 consensus_queue.cc:260] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.209674 11699 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:24.210323 11699 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.213095 11699 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:24.213528 11424 ts_tablet_manager.cc:1434] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260321 02:58:24.214548 11424 tablet_bootstrap.cc:492] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:24.217293 11425 ts_tablet_manager.cc:1434] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.048s user 0.007s sys 0.000s
I20260321 02:58:24.218916 11425 tablet_bootstrap.cc:492] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:24.220618 11424 tablet_bootstrap.cc:654] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.202777 11429 ts_tablet_manager.cc:1434] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260321 02:58:24.221810 11429 tablet_bootstrap.cc:492] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:24.225426 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:24.226267 11307 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:24.232197 11425 tablet_bootstrap.cc:654] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.234299 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:24.240422 11382 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:24.232955 11130 catalog_manager.cc:5671] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 2, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 2 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:24.236609 11200 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:24.239398 11429 tablet_bootstrap.cc:654] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.253491 11699 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:24.253981 11699 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:24.254383 11699 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.272903 11425 tablet_bootstrap.cc:492] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:24.295166 11425 ts_tablet_manager.cc:1403] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.076s user 0.019s sys 0.000s
I20260321 02:58:24.294622 11429 tablet_bootstrap.cc:492] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:24.296926 11429 ts_tablet_manager.cc:1403] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.075s user 0.014s sys 0.006s
I20260321 02:58:24.296627 11699 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.298734 11699 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:24.302712 11424 tablet_bootstrap.cc:492] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:24.302515 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:24.303202 11424 ts_tablet_manager.cc:1403] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.089s user 0.033s sys 0.025s
I20260321 02:58:24.303215 11307 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.301326 11429 raft_consensus.cc:359] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.314900 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:24.315554 11379 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.315915 11424 raft_consensus.cc:359] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.314895 11429 raft_consensus.cc:385] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.311508 11307 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:24.319929 11429 raft_consensus.cc:740] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.303668 11425 raft_consensus.cc:359] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.324251 11425 raft_consensus.cc:385] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.324563 11425 raft_consensus.cc:740] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.324663 11429 consensus_queue.cc:260] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.325179 11425 consensus_queue.cc:260] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.316601 11424 raft_consensus.cc:385] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.327294 11429 ts_tablet_manager.cc:1434] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.029s user 0.003s sys 0.003s
I20260321 02:58:24.327450 11424 raft_consensus.cc:740] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.321343 11200 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:24.328405 11699 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:24.328280 11424 consensus_queue.cc:260] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.328960 11699 raft_consensus.cc:697] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:24.318483 11130 catalog_manager.cc:5671] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:24.329779 11699 consensus_queue.cc:237] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.330677 11424 ts_tablet_manager.cc:1434] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.026s user 0.014s sys 0.000s
I20260321 02:58:24.322907 11379 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:24.312937 11691 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:24.331327 11691 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.333317 11691 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:24.327306 11425 ts_tablet_manager.cc:1434] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.032s user 0.003s sys 0.000s
I20260321 02:58:24.335335 11425 tablet_bootstrap.cc:492] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:24.335896 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:24.336573 11230 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
I20260321 02:58:24.339668 11275 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:24.340447 11691 raft_consensus.cc:2804] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:24.340802 11691 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:24.341120 11691 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.336580 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:58:24.344609 11275 leader_election.cc:343] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:24.353052 11424 tablet_bootstrap.cc:492] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:24.358479 11425 tablet_bootstrap.cc:654] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.359534 11424 tablet_bootstrap.cc:654] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.362190 11132 catalog_manager.cc:5671] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:24.365676 11691 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.366487 11429 tablet_bootstrap.cc:492] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:24.371608 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:24.372243 11230 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.381330 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:24.382784 11230 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
W20260321 02:58:24.383572 11275 leader_election.cc:343] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:24.386888 11275 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:24.387743 11684 raft_consensus.cc:2804] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:24.388063 11691 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:24.388566 11684 raft_consensus.cc:697] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:58:24.391086 11429 tablet_bootstrap.cc:654] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.390304 11684 consensus_queue.cc:237] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.396984 11424 tablet_bootstrap.cc:492] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:24.397430 11425 tablet_bootstrap.cc:492] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:24.397543 11424 ts_tablet_manager.cc:1403] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.045s user 0.010s sys 0.023s
I20260321 02:58:24.398111 11425 ts_tablet_manager.cc:1403] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.063s user 0.010s sys 0.021s
I20260321 02:58:24.400701 11425 raft_consensus.cc:359] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.400954 11424 raft_consensus.cc:359] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.401849 11424 raft_consensus.cc:385] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.402186 11424 raft_consensus.cc:740] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.401419 11425 raft_consensus.cc:385] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.403230 11425 raft_consensus.cc:740] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.404102 11425 consensus_queue.cc:260] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.408733 11549 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:24.409276 11549 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.411327 11549 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:24.412398 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:24.408160 11425 ts_tablet_manager.cc:1434] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260321 02:58:24.413118 11307 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:24.413378 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:24.414045 11425 tablet_bootstrap.cc:492] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:24.414048 11230 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:24.414456 11349 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:24.414829 11429 tablet_bootstrap.cc:492] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:24.415221 11429 ts_tablet_manager.cc:1403] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.076s user 0.016s sys 0.004s
I20260321 02:58:24.415704 11695 raft_consensus.cc:2804] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:24.416611 11695 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:24.416980 11695 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.418514 11429 raft_consensus.cc:359] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.419315 11429 raft_consensus.cc:385] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.419616 11429 raft_consensus.cc:740] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.420321 11429 consensus_queue.cc:260] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.402880 11424 consensus_queue.cc:260] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.422817 11424 ts_tablet_manager.cc:1434] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20260321 02:58:24.424033 11424 tablet_bootstrap.cc:492] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:24.429641 11429 ts_tablet_manager.cc:1434] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.014s user 0.010s sys 0.000s
I20260321 02:58:24.430645 11429 tablet_bootstrap.cc:492] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:24.431439 11424 tablet_bootstrap.cc:654] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.437652 11549 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:24.438197 11549 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.439738 11429 tablet_bootstrap.cc:654] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.440147 11549 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:24.441641 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:24.444284 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:24.446458 11699 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:24.446966 11699 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:24.448192 11349 leader_election.cc:343] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:24.449272 11699 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:58:24.451171 11349 leader_election.cc:343] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:24.452313 11349 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:24.415692 11132 catalog_manager.cc:5671] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:24.454558 11429 tablet_bootstrap.cc:492] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:24.455572 11429 ts_tablet_manager.cc:1403] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.008s
I20260321 02:58:24.456281 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:24.457067 11379 raft_consensus.cc:2393] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 1: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:24.458006 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:24.458076 11429 raft_consensus.cc:359] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.458948 11429 raft_consensus.cc:385] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.459210 11429 raft_consensus.cc:740] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.458155 11695 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.459803 11200 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:24.460477 11694 raft_consensus.cc:2749] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:24.460891 11709 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.461285 11695 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:24.459846 11429 consensus_queue.cc:260] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.462790 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:24.463831 11307 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.466180 11429 ts_tablet_manager.cc:1434] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.010s user 0.008s sys 0.001s
I20260321 02:58:24.467449 11429 tablet_bootstrap.cc:492] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:24.466333 11425 tablet_bootstrap.cc:654] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.471179 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:24.471825 11230 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.475088 11429 tablet_bootstrap.cc:654] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.484405 11307 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:24.486266 11709 raft_consensus.cc:2749] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:24.487751 11349 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:24.490657 11549 raft_consensus.cc:2804] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:24.491863 11549 raft_consensus.cc:697] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:24.493892 11425 tablet_bootstrap.cc:492] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:24.495743 11230 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:24.496008 11425 ts_tablet_manager.cc:1403] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.082s user 0.014s sys 0.011s
I20260321 02:58:24.492795 11549 consensus_queue.cc:237] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.497396 11684 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:24.498067 11684 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.500452 11424 tablet_bootstrap.cc:492] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:24.501031 11424 ts_tablet_manager.cc:1403] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.077s user 0.020s sys 0.023s
I20260321 02:58:24.501029 11425 raft_consensus.cc:359] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.501443 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:24.501865 11425 raft_consensus.cc:385] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.500394 11684 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:24.502233 11425 raft_consensus.cc:740] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
W20260321 02:58:24.503050 11275 leader_election.cc:343] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:24.503171 11425 consensus_queue.cc:260] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.504779 11424 raft_consensus.cc:359] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.504956 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:24.505700 11424 raft_consensus.cc:385] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.505725 11429 tablet_bootstrap.cc:492] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:24.506181 11424 raft_consensus.cc:740] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.506765 11429 ts_tablet_manager.cc:1403] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.040s user 0.015s sys 0.009s
W20260321 02:58:24.506645 11275 leader_election.cc:343] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:24.507262 11275 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:24.508127 11684 raft_consensus.cc:2749] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:24.508560 11425 ts_tablet_manager.cc:1434] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260321 02:58:24.507232 11424 consensus_queue.cc:260] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.509689 11425 tablet_bootstrap.cc:492] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:24.511502 11424 ts_tablet_manager.cc:1434] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260321 02:58:24.515118 11424 tablet_bootstrap.cc:492] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:24.517134 11425 tablet_bootstrap.cc:654] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.523416 11424 tablet_bootstrap.cc:654] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.512502 11429 raft_consensus.cc:359] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.526572 11429 raft_consensus.cc:385] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.527159 11429 raft_consensus.cc:740] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.529017 11429 consensus_queue.cc:260] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.529894 11133 catalog_manager.cc:5671] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:24.531137 11537 consensus_queue.cc:1048] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:24.537994 11425 tablet_bootstrap.cc:492] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:24.538513 11425 ts_tablet_manager.cc:1403] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.008s
I20260321 02:58:24.541464 11425 raft_consensus.cc:359] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.542649 11425 raft_consensus.cc:385] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.542925 11425 raft_consensus.cc:740] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.543517 11425 consensus_queue.cc:260] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.544880 11429 ts_tablet_manager.cc:1434] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.038s user 0.008s sys 0.000s
I20260321 02:58:24.545811 11425 ts_tablet_manager.cc:1434] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260321 02:58:24.546655 11425 tablet_bootstrap.cc:492] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:24.552882 11425 tablet_bootstrap.cc:654] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.558542 11429 tablet_bootstrap.cc:492] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:24.577378 11537 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:24.577934 11537 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.598655 11424 tablet_bootstrap.cc:492] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:24.600102 11699 consensus_queue.cc:1048] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:24.603767 11424 ts_tablet_manager.cc:1403] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.089s user 0.014s sys 0.026s
I20260321 02:58:24.608160 11691 consensus_queue.cc:1048] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:24.608503 11537 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:24.615410 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:24.622751 11228 raft_consensus.cc:1275] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260321 02:58:24.625206 11228 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.630141 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:24.631217 11379 raft_consensus.cc:2393] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 1: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:24.633356 11694 consensus_queue.cc:1048] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:24.634843 11275 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:24.635730 11537 raft_consensus.cc:3060] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.638473 11425 tablet_bootstrap.cc:492] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:24.638945 11694 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:24.639397 11425 ts_tablet_manager.cc:1403] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.093s user 0.011s sys 0.020s
I20260321 02:58:24.639416 11694 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.641871 11425 raft_consensus.cc:359] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.642552 11425 raft_consensus.cc:385] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.643021 11425 raft_consensus.cc:740] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.648132 11429 tablet_bootstrap.cc:654] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.628089 11424 raft_consensus.cc:359] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.650569 11424 raft_consensus.cc:385] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.651269 11424 raft_consensus.cc:740] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.652650 11424 consensus_queue.cc:260] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.653605 11694 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:24.643821 11425 consensus_queue.cc:260] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.663594 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:24.665398 11235 raft_consensus.cc:2393] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 1: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:24.665781 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:24.668491 11425 ts_tablet_manager.cc:1434] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.029s user 0.008s sys 0.000s
I20260321 02:58:24.670361 11424 ts_tablet_manager.cc:1434] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.066s user 0.010s sys 0.000s
I20260321 02:58:24.671545 11424 tablet_bootstrap.cc:492] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:24.672363 11425 tablet_bootstrap.cc:492] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:24.678756 11425 tablet_bootstrap.cc:654] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.683542 11424 tablet_bootstrap.cc:654] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.693040 11349 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:24.694449 11425 tablet_bootstrap.cc:492] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:24.695124 11425 ts_tablet_manager.cc:1403] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.004s
I20260321 02:58:24.695147 11549 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.699291 11425 raft_consensus.cc:359] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.707873 11425 raft_consensus.cc:385] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.708089 11424 tablet_bootstrap.cc:492] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:24.708461 11425 raft_consensus.cc:740] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.708964 11424 ts_tablet_manager.cc:1403] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.038s user 0.022s sys 0.008s
I20260321 02:58:24.709769 11425 consensus_queue.cc:260] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.711714 11425 ts_tablet_manager.cc:1434] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.016s user 0.006s sys 0.003s
I20260321 02:58:24.711719 11424 raft_consensus.cc:359] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.712636 11424 raft_consensus.cc:385] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.712751 11425 tablet_bootstrap.cc:492] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:24.713009 11424 raft_consensus.cc:740] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.713611 11424 consensus_queue.cc:260] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.715878 11424 ts_tablet_manager.cc:1434] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20260321 02:58:24.717391 11424 tablet_bootstrap.cc:492] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:24.722259 11537 raft_consensus.cc:2749] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:24.723402 11424 tablet_bootstrap.cc:654] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.768682 11425 tablet_bootstrap.cc:654] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.779179 11694 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:24.786865 11694 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.788842 11694 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:24.796809 11694 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:24.797314 11694 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.770509 11709 consensus_queue.cc:1048] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20260321 02:58:24.801394 11694 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:24.810213 11549 raft_consensus.cc:2749] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:24.824064 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:24.824985 11228 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:24.826195 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:24.829453 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:24.825069 11309 raft_consensus.cc:1275] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260321 02:58:24.830915 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:24.831647 11307 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:24.837952 11349 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:24.838882 11429 tablet_bootstrap.cc:492] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:24.839380 11429 ts_tablet_manager.cc:1403] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.281s user 0.034s sys 0.051s
I20260321 02:58:24.840368 11694 raft_consensus.cc:2804] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260321 02:58:24.840442 11349 leader_election.cc:343] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:24.840762 11694 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:24.841979 11694 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.846848 11429 raft_consensus.cc:359] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.847622 11429 raft_consensus.cc:385] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.848094 11429 raft_consensus.cc:740] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.848781 11429 consensus_queue.cc:260] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.849879 11424 tablet_bootstrap.cc:492] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:24.850360 11424 ts_tablet_manager.cc:1403] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.133s user 0.028s sys 0.022s
I20260321 02:58:24.851641 11429 ts_tablet_manager.cc:1434] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
W20260321 02:58:24.841595 11349 leader_election.cc:343] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:24.853953 11349 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:24.855182 11695 raft_consensus.cc:2749] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:24.855803 11695 consensus_queue.cc:1048] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:24.857097 11429 tablet_bootstrap.cc:492] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:24.858481 11694 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.860744 11694 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:24.868285 11424 raft_consensus.cc:359] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.869163 11424 raft_consensus.cc:385] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.869628 11424 raft_consensus.cc:740] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.869828 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:24.868351 11425 tablet_bootstrap.cc:492] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:24.870468 11235 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.870630 11425 ts_tablet_manager.cc:1403] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.158s user 0.010s sys 0.010s
I20260321 02:58:24.873101 11425 raft_consensus.cc:359] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.876400 11424 consensus_queue.cc:260] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.878692 11709 consensus_queue.cc:1048] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:24.880013 11235 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:24.880821 11429 tablet_bootstrap.cc:654] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.883592 11424 ts_tablet_manager.cc:1434] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.033s user 0.008s sys 0.000s
I20260321 02:58:24.885195 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:24.885809 11307 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.886651 11349 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:24.873749 11425 raft_consensus.cc:385] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.888322 11764 raft_consensus.cc:2804] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:24.888465 11425 raft_consensus.cc:740] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.890763 11764 raft_consensus.cc:697] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:24.890753 11425 consensus_queue.cc:260] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.892095 11424 tablet_bootstrap.cc:492] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:24.891780 11764 consensus_queue.cc:237] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.889114 11709 consensus_queue.cc:1048] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260321 02:58:24.901721 11695 consensus_queue.cc:1048] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:24.907862 11307 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:24.912627 11699 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:24.913295 11699 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.915352 11699 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:24.918066 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:24.918769 11306 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:24.920619 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:24.924046 11200 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:24.924751 11425 ts_tablet_manager.cc:1434] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.054s user 0.006s sys 0.001s
I20260321 02:58:24.925735 11425 tablet_bootstrap.cc:492] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:24.925899 11702 raft_consensus.cc:2804] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:24.926256 11702 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:24.926605 11702 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.929761 11537 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:24.930280 11537 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.932163 11537 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:24.932554 11684 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:24.933040 11684 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.933434 11695 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:24.933682 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:24.933895 11695 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.934921 11684 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:58:24.935072 11275 leader_election.cc:343] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:24.935859 11695 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:24.936549 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:24.937222 11235 raft_consensus.cc:2468] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
I20260321 02:58:24.937723 11424 tablet_bootstrap.cc:654] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.938148 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:24.938408 11275 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:24.938830 11383 raft_consensus.cc:2468] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
I20260321 02:58:24.939595 11684 raft_consensus.cc:2804] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:24.939707 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:24.940047 11684 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:24.940416 11684 raft_consensus.cc:3060] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.942795 11702 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.944881 11702 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:24.947330 11684 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.951185 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:24.951875 11230 raft_consensus.cc:3060] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.953702 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:24.954282 11310 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.955825 11425 tablet_bootstrap.cc:654] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:24.960026 11230 raft_consensus.cc:2468] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:24.961302 11310 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:24.961509 11275 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:24.962514 11537 raft_consensus.cc:2804] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:24.963086 11537 raft_consensus.cc:697] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:58:24.963454 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:24.964097 11537 consensus_queue.cc:237] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.965054 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:24.965677 11379 raft_consensus.cc:2468] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
I20260321 02:58:24.955973 11429 tablet_bootstrap.cc:492] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:24.969408 11429 ts_tablet_manager.cc:1403] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.113s user 0.012s sys 0.000s
W20260321 02:58:24.972576 11200 leader_election.cc:343] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:24.973930 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:58:24.974253 11349 leader_election.cc:343] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:24.974618 11235 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:24.975260 11275 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:24.976438 11691 raft_consensus.cc:2804] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:24.976886 11691 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:24.977450 11691 raft_consensus.cc:3060] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
W20260321 02:58:24.978170 11200 leader_election.cc:343] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:24.984323 11709 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:24.984871 11709 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.987084 11709 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:24.990365 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:24.990998 11379 raft_consensus.cc:3060] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:24.971191 11684 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:24.996239 11429 raft_consensus.cc:359] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:24.996969 11429 raft_consensus.cc:385] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:24.997437 11429 raft_consensus.cc:740] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:24.998140 11429 consensus_queue.cc:260] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.000039 11349 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:25.000490 11429 ts_tablet_manager.cc:1434] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.030s user 0.007s sys 0.000s
I20260321 02:58:25.001576 11429 tablet_bootstrap.cc:492] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:25.001652 11200 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:25.002367 11709 raft_consensus.cc:2804] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:25.002833 11709 raft_consensus.cc:697] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:24.991417 11691 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.003666 11709 consensus_queue.cc:237] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.005316 11691 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:25.009559 11694 raft_consensus.cc:2804] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:25.010043 11694 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:25.010425 11694 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:25.013680 11684 consensus_queue.cc:1048] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:25.016192 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:25.020790 11537 consensus_queue.cc:1048] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260321 02:58:25.021869 11275 leader_election.cc:343] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:25.023022 11379 raft_consensus.cc:2468] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:25.025789 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:25.026408 11383 raft_consensus.cc:3060] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:25.033535 11383 raft_consensus.cc:2468] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:25.034760 11424 tablet_bootstrap.cc:492] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:25.035269 11424 ts_tablet_manager.cc:1403] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.151s user 0.025s sys 0.011s
I20260321 02:58:25.035307 11425 tablet_bootstrap.cc:492] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:25.036113 11425 ts_tablet_manager.cc:1403] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.111s user 0.011s sys 0.026s
I20260321 02:58:25.039438 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:25.040838 11134 catalog_manager.cc:5671] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:25.021265 11694 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.048329 11425 raft_consensus.cc:359] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.107826 11425 raft_consensus.cc:385] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:25.108331 11425 raft_consensus.cc:740] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:25.109117 11425 consensus_queue.cc:260] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.048472 11309 raft_consensus.cc:2468] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
W20260321 02:58:25.103346 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 46289.8us
I20260321 02:58:25.105808 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:25.092962 11429 tablet_bootstrap.cc:654] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:25.109514 11694 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:25.055497 11424 raft_consensus.cc:359] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.111671 11424 raft_consensus.cc:385] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:25.112008 11424 raft_consensus.cc:740] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:25.115813 11424 consensus_queue.cc:260] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.117228 11425 ts_tablet_manager.cc:1434] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.081s user 0.010s sys 0.009s
I20260321 02:58:25.118542 11424 ts_tablet_manager.cc:1434] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.083s user 0.013s sys 0.000s
I20260321 02:58:25.118839 11425 tablet_bootstrap.cc:492] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:25.119643 11424 tablet_bootstrap.cc:492] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:25.120980 11275 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:25.128774 11537 raft_consensus.cc:2804] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:25.129302 11537 raft_consensus.cc:697] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:58:25.130113 11537 consensus_queue.cc:237] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.135982 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:25.136827 11429 tablet_bootstrap.cc:492] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:25.137506 11429 ts_tablet_manager.cc:1403] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.136s user 0.015s sys 0.000s
I20260321 02:58:25.123066 11200 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:25.123881 11764 consensus_queue.cc:1048] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:25.143471 11425 tablet_bootstrap.cc:654] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
W20260321 02:58:25.150956 11182 auto_rebalancer.cc:247] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260321 02:58:25.155030 11349 leader_election.cc:343] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:25.158651 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:25.159188 11235 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
W20260321 02:58:25.163800 11200 leader_election.cc:343] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:25.157622 11709 raft_consensus.cc:2804] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:25.169366 11709 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:25.181497 11235 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:25.183583 11709 raft_consensus.cc:3060] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:25.193944 11429 raft_consensus.cc:359] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.194725 11429 raft_consensus.cc:385] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:25.194989 11429 raft_consensus.cc:740] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:25.195590 11429 consensus_queue.cc:260] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.197472 11781 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:58:25.234925 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 69629.2us
I20260321 02:58:25.237174 11429 ts_tablet_manager.cc:1434] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.099s user 0.008s sys 0.000s
I20260321 02:58:25.238960 11349 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:25.242599 11764 raft_consensus.cc:2804] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:25.228799 11781 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.250618 11764 raft_consensus.cc:697] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:25.256841 11764 consensus_queue.cc:237] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.257901 11781 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:25.261513 11429 tablet_bootstrap.cc:492] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:25.268440 11424 tablet_bootstrap.cc:654] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:25.283638 11709 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.285749 11709 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:25.297823 11130 catalog_manager.cc:5671] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
W20260321 02:58:25.331035 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 92141.3us
I20260321 02:58:25.361307 11133 catalog_manager.cc:5671] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
W20260321 02:58:25.373940 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 132603us
I20260321 02:58:25.375882 11134 catalog_manager.cc:5671] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:25.404572 11789 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:25.405150 11789 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.407549 11789 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:25.400818 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:25.413426 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:25.418170 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:25.418792 11307 raft_consensus.cc:3060] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
W20260321 02:58:25.430410 11349 leader_election.cc:343] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:25.434511 11781 consensus_queue.cc:1048] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260321 02:58:25.427034 11800 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:25.438968 11800 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.444819 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:25.454281 11429 tablet_bootstrap.cc:654] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:25.455241 11800 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:25.460836 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:25.461359 11130 catalog_manager.cc:5671] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
W20260321 02:58:25.475842 11200 leader_election.cc:343] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:25.484639 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:25.484820 11637 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:25.485796 11425 tablet_bootstrap.cc:492] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:25.486006 11308 raft_consensus.cc:2393] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 1: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:25.485879 11637 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.486654 11777 consensus_queue.cc:1048] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:25.488039 11637 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:25.492718 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:25.486393 11425 ts_tablet_manager.cc:1403] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.368s user 0.014s sys 0.023s
I20260321 02:58:25.498615 11382 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:25.498778 11796 consensus_queue.cc:1048] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260321 02:58:25.497522 11349 leader_election.cc:343] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:25.500633 11424 tablet_bootstrap.cc:492] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:25.500404 11349 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:25.501132 11424 ts_tablet_manager.cc:1403] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.382s user 0.049s sys 0.049s
I20260321 02:58:25.501422 11764 raft_consensus.cc:2749] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:25.505044 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:58:25.507254 11275 leader_election.cc:343] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:25.507261 11383 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
I20260321 02:58:25.509810 11275 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:25.511135 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:25.512240 11794 raft_consensus.cc:2804] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:25.512598 11794 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:25.512929 11794 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:25.505424 11424 raft_consensus.cc:359] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.505947 11425 raft_consensus.cc:359] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.515026 11424 raft_consensus.cc:385] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:25.515203 11425 raft_consensus.cc:385] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:25.515319 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:25.515502 11425 raft_consensus.cc:740] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:25.515348 11424 raft_consensus.cc:740] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:25.516238 11383 raft_consensus.cc:2393] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 1: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:25.516309 11425 consensus_queue.cc:260] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.516345 11424 consensus_queue.cc:260] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.528772 11789 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:25.530515 11789 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.534991 11425 ts_tablet_manager.cc:1434] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20260321 02:58:25.527051 11307 raft_consensus.cc:2468] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:25.539778 11349 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:25.541008 11695 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:25.577689 11695 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.579031 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:25.573644 11425 tablet_bootstrap.cc:492] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:25.539312 11424 ts_tablet_manager.cc:1434] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.038s user 0.007s sys 0.000s
I20260321 02:58:25.582394 11424 tablet_bootstrap.cc:492] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:25.582755 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:25.583453 11379 raft_consensus.cc:2468] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
I20260321 02:58:25.584486 11695 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:58:25.587157 11275 leader_election.cc:343] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:25.582064 11200 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:25.589434 11777 consensus_queue.cc:1048] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:25.590428 11709 raft_consensus.cc:3060] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:25.590997 11200 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:25.592360 11637 raft_consensus.cc:2804] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:25.594017 11637 raft_consensus.cc:697] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:25.591066 11275 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:25.594966 11637 consensus_queue.cc:237] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.604625 11425 tablet_bootstrap.cc:654] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:25.534386 11789 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:25.605412 11796 consensus_queue.cc:1048] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:25.600374 11805 consensus_queue.cc:1048] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260321 02:58:25.600571 11782 raft_consensus.cc:2749] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:25.612461 11777 raft_consensus.cc:2804] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:25.612905 11424 tablet_bootstrap.cc:654] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:25.612990 11777 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:25.613554 11777 raft_consensus.cc:3060] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:25.614477 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:25.622854 11794 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:25.630630 11349 leader_election.cc:343] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:25.646157 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:25.625434 11709 raft_consensus.cc:2749] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:25.664703 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:25.664763 11429 tablet_bootstrap.cc:492] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:25.666416 11429 ts_tablet_manager.cc:1403] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.405s user 0.017s sys 0.016s
I20260321 02:58:25.653586 11794 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:58:25.680796 11275 leader_election.cc:343] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:25.679642 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:25.695827 11382 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:25.689050 11777 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.713310 11306 raft_consensus.cc:2393] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 1: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:25.714133 11777 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:25.717239 11429 raft_consensus.cc:359] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.718300 11429 raft_consensus.cc:385] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:25.720264 11429 raft_consensus.cc:740] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:25.723395 11794 consensus_queue.cc:1048] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260321 02:58:25.725147 11807 log.cc:927] Time spent T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
W20260321 02:58:25.728295 11349 consensus_peers.cc:597] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 -> Peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Couldn't send request to peer 83b12b883a7f4129a87dadf5a78f08da. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:25.721452 11429 consensus_queue.cc:260] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.729671 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:25.729921 11349 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:25.730242 11385 raft_consensus.cc:3060] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:25.733731 11424 tablet_bootstrap.cc:492] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:25.734810 11424 ts_tablet_manager.cc:1403] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.153s user 0.011s sys 0.040s
I20260321 02:58:25.735170 11425 tablet_bootstrap.cc:492] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:25.735688 11425 ts_tablet_manager.cc:1403] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.162s user 0.014s sys 0.026s
I20260321 02:58:25.758690 11429 ts_tablet_manager.cc:1434] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.092s user 0.009s sys 0.000s
I20260321 02:58:25.759645 11429 tablet_bootstrap.cc:492] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
W20260321 02:58:25.755599 11810 log.cc:927] Time spent T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.103s user 0.000s sys 0.002s
I20260321 02:58:25.754492 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:25.794549 11385 raft_consensus.cc:2468] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:25.825907 11382 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:25.796489 11764 raft_consensus.cc:2749] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:25.837981 11429 tablet_bootstrap.cc:654] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:25.856829 11424 raft_consensus.cc:359] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.858076 11424 raft_consensus.cc:385] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:25.858397 11424 raft_consensus.cc:740] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:25.859123 11424 consensus_queue.cc:260] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.862391 11425 raft_consensus.cc:359] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.863479 11425 raft_consensus.cc:385] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:25.863873 11425 raft_consensus.cc:740] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:25.864805 11425 consensus_queue.cc:260] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.875833 11424 ts_tablet_manager.cc:1434] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.141s user 0.036s sys 0.040s
W20260321 02:58:25.870766 11776 log.cc:927] Time spent T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.072s user 0.007s sys 0.005s
W20260321 02:58:25.881912 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 123774us
I20260321 02:58:25.870407 11796 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:25.884588 11275 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:25.887406 11275 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:25.885758 11424 tablet_bootstrap.cc:492] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:25.888353 11794 raft_consensus.cc:2804] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:25.889703 11794 raft_consensus.cc:697] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:58:25.888680 11425 ts_tablet_manager.cc:1434] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.153s user 0.009s sys 0.000s
I20260321 02:58:25.888499 11796 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.890573 11794 consensus_queue.cc:237] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.891480 11814 raft_consensus.cc:2804] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:25.909536 11814 raft_consensus.cc:697] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
W20260321 02:58:25.896088 11275 leader_election.cc:343] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:25.900274 11425 tablet_bootstrap.cc:492] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:25.901170 11133 catalog_manager.cc:5671] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:25.906069 11796 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:25.918174 11819 consensus_queue.cc:1048] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:25.910981 11814 consensus_queue.cc:237] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.926126 11764 consensus_queue.cc:1048] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:25.927212 11424 tablet_bootstrap.cc:654] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:25.937420 11425 tablet_bootstrap.cc:654] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:25.938253 11424 tablet_bootstrap.cc:492] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:25.938737 11424 ts_tablet_manager.cc:1403] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.062s user 0.013s sys 0.001s
I20260321 02:58:25.942148 11424 raft_consensus.cc:359] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.942833 11424 raft_consensus.cc:385] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:25.943102 11424 raft_consensus.cc:740] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:25.943712 11424 consensus_queue.cc:260] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.945878 11424 ts_tablet_manager.cc:1434] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260321 02:58:25.951740 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:25.955718 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:25.973991 11429 tablet_bootstrap.cc:492] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:25.974893 11429 ts_tablet_manager.cc:1403] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.215s user 0.024s sys 0.009s
I20260321 02:58:25.977428 11424 tablet_bootstrap.cc:492] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
W20260321 02:58:25.979316 11200 leader_election.cc:343] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:25.990769 11788 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:25.991302 11788 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:25.993230 11788 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:25.995186 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:58:25.996732 11275 leader_election.cc:343] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
W20260321 02:58:26.001554 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 44272.3us
W20260321 02:58:26.016417 11200 leader_election.cc:343] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:26.019145 11429 raft_consensus.cc:359] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.020118 11429 raft_consensus.cc:385] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:26.020445 11429 raft_consensus.cc:740] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:26.021138 11429 consensus_queue.cc:260] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.021929 11200 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:26.025352 11796 raft_consensus.cc:2749] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:26.040235 11425 tablet_bootstrap.cc:492] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:26.041319 11425 ts_tablet_manager.cc:1403] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.141s user 0.033s sys 0.039s
I20260321 02:58:26.042261 11429 ts_tablet_manager.cc:1434] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.067s user 0.005s sys 0.002s
I20260321 02:58:26.043807 11429 tablet_bootstrap.cc:492] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:26.044333 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:26.047263 11425 raft_consensus.cc:359] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.048135 11425 raft_consensus.cc:385] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:26.048388 11425 raft_consensus.cc:740] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:26.053596 11425 consensus_queue.cc:260] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.055375 11814 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:26.055830 11425 ts_tablet_manager.cc:1434] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20260321 02:58:26.055876 11814 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.057060 11425 tablet_bootstrap.cc:492] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:26.057893 11814 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:26.061299 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:26.063880 11424 tablet_bootstrap.cc:654] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
W20260321 02:58:26.068503 11275 leader_election.cc:343] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:26.068953 11275 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:26.069932 11814 raft_consensus.cc:2749] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:26.072794 11425 tablet_bootstrap.cc:654] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
W20260321 02:58:26.076095 11275 leader_election.cc:343] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:26.077667 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:26.081892 11424 tablet_bootstrap.cc:492] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:26.082410 11424 ts_tablet_manager.cc:1403] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.128s user 0.014s sys 0.007s
I20260321 02:58:26.084985 11429 tablet_bootstrap.cc:654] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:26.085001 11424 raft_consensus.cc:359] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.085870 11424 raft_consensus.cc:385] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:26.086145 11424 raft_consensus.cc:740] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:26.087112 11424 consensus_queue.cc:260] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.089862 11425 tablet_bootstrap.cc:492] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:26.090397 11425 ts_tablet_manager.cc:1403] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.005s
I20260321 02:58:26.090823 11424 ts_tablet_manager.cc:1434] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
W20260321 02:58:26.143100 11275 leader_election.cc:343] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:26.144665 11275 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:26.091765 11424 tablet_bootstrap.cc:492] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:26.160831 11709 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:26.162195 11709 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.165586 11814 raft_consensus.cc:2749] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:26.178321 11381 raft_consensus.cc:3060] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:26.196369 11425 raft_consensus.cc:359] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.197443 11425 raft_consensus.cc:385] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:26.197826 11425 raft_consensus.cc:740] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:26.210892 11429 tablet_bootstrap.cc:492] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:26.211763 11429 ts_tablet_manager.cc:1403] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.168s user 0.016s sys 0.004s
I20260321 02:58:26.226080 11709 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:26.220403 11425 consensus_queue.cc:260] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.262277 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:26.262415 11134 catalog_manager.cc:5671] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
W20260321 02:58:26.265336 11200 leader_election.cc:343] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:26.273828 11429 raft_consensus.cc:359] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.277838 11429 raft_consensus.cc:385] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:26.278216 11429 raft_consensus.cc:740] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:26.278795 11429 consensus_queue.cc:260] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.280104 11134 catalog_manager.cc:5671] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:26.284062 11425 ts_tablet_manager.cc:1434] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.193s user 0.012s sys 0.006s
I20260321 02:58:26.289494 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:26.290098 11424 tablet_bootstrap.cc:654] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:26.290357 11383 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:26.293783 11429 ts_tablet_manager.cc:1434] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.081s user 0.005s sys 0.002s
I20260321 02:58:26.296237 11781 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:26.296334 11429 tablet_bootstrap.cc:492] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:26.296859 11781 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.297479 11200 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:26.303598 11709 raft_consensus.cc:2804] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:26.304034 11709 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:26.304421 11709 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:26.310928 11702 consensus_queue.cc:1048] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:26.320269 11699 consensus_queue.cc:1048] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:26.324442 11709 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.327008 11709 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:26.327483 11425 tablet_bootstrap.cc:492] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:26.332717 11797 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:26.333560 11797 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.338027 11797 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:26.343811 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:26.345387 11781 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:26.352207 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:26.359066 11425 tablet_bootstrap.cc:654] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:26.365813 11424 tablet_bootstrap.cc:492] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:26.366858 11383 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:26.367367 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:26.373159 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:58:26.411492 11200 leader_election.cc:343] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:26.414537 11424 ts_tablet_manager.cc:1403] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.323s user 0.022s sys 0.004s
I20260321 02:58:26.421856 11429 tablet_bootstrap.cc:654] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:26.427284 11424 raft_consensus.cc:359] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.430595 11424 raft_consensus.cc:385] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:26.431005 11424 raft_consensus.cc:740] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:26.430881 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:26.431784 11424 consensus_queue.cc:260] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.432322 11699 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:26.432799 11699 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.433678 11424 ts_tablet_manager.cc:1434] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260321 02:58:26.435248 11424 tablet_bootstrap.cc:492] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:26.435107 11699 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:26.439929 11383 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:26.437458 11236 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:26.440472 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:26.465849 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:26.466748 11385 raft_consensus.cc:2393] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 1: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
W20260321 02:58:26.467123 11349 leader_election.cc:343] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:26.471799 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:58:26.472898 11275 consensus_peers.cc:597] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da -> Peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Couldn't send request to peer 0d7efc590a1041a9bd4cd4a971332dfb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:26.473021 11200 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:26.473256 11232 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:26.474491 11831 raft_consensus.cc:2804] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:26.474018 11349 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:26.476029 11694 raft_consensus.cc:2749] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:26.477124 11831 raft_consensus.cc:697] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:26.478143 11831 consensus_queue.cc:237] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:26.479314 11182 auto_rebalancer.cc:247] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260321 02:58:26.486325 11425 tablet_bootstrap.cc:492] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:26.486881 11425 ts_tablet_manager.cc:1403] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.160s user 0.027s sys 0.009s
I20260321 02:58:26.488227 11200 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:26.496438 11699 raft_consensus.cc:2749] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:26.497076 11425 raft_consensus.cc:359] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.497835 11425 raft_consensus.cc:385] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:26.498142 11425 raft_consensus.cc:740] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:26.498817 11425 consensus_queue.cc:260] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.505740 11788 consensus_queue.cc:1048] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260321 02:58:26.509339 11777 consensus_queue.cc:1048] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:26.550453 11425 ts_tablet_manager.cc:1434] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.063s user 0.005s sys 0.003s
I20260321 02:58:26.551527 11424 tablet_bootstrap.cc:654] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:26.551640 11425 tablet_bootstrap.cc:492] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:26.552584 11429 tablet_bootstrap.cc:492] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:26.553045 11429 ts_tablet_manager.cc:1403] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.258s user 0.007s sys 0.033s
I20260321 02:58:26.563604 11349 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:26.565414 11781 raft_consensus.cc:2804] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:26.566193 11781 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:26.566704 11781 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:26.575102 11781 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.585116 11788 consensus_queue.cc:1048] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:26.593142 11764 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:26.594399 11764 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.595585 11134 catalog_manager.cc:5671] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:26.598119 11429 raft_consensus.cc:359] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.598840 11429 raft_consensus.cc:385] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:26.599102 11429 raft_consensus.cc:740] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:26.596467 11764 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:26.605902 11429 consensus_queue.cc:260] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.610323 11425 tablet_bootstrap.cc:654] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:26.614368 11424 tablet_bootstrap.cc:492] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:26.614886 11424 ts_tablet_manager.cc:1403] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.180s user 0.011s sys 0.001s
I20260321 02:58:26.615271 11781 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:26.616937 11429 ts_tablet_manager.cc:1434] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.064s user 0.007s sys 0.002s
I20260321 02:58:26.619944 11429 tablet_bootstrap.cc:492] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:26.625415 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:26.628803 11424 raft_consensus.cc:359] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.630414 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:26.631254 11230 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:26.630419 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:26.633533 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:26.633744 11306 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:26.642652 11230 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:26.629855 11424 raft_consensus.cc:385] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:26.643791 11424 raft_consensus.cc:740] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:26.644547 11424 consensus_queue.cc:260] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:26.649293 11349 leader_election.cc:343] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:26.653066 11349 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:26.655020 11841 raft_consensus.cc:2804] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:26.655435 11841 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:26.655771 11841 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:26.657469 11794 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:26.657966 11794 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.660281 11794 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:26.661226 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:26.662907 11425 tablet_bootstrap.cc:492] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:26.663398 11425 ts_tablet_manager.cc:1403] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.112s user 0.019s sys 0.000s
I20260321 02:58:26.663652 11429 tablet_bootstrap.cc:654] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
W20260321 02:58:26.664973 11349 leader_election.cc:343] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:26.670713 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:26.671748 11841 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.674070 11841 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:26.678356 11275 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:26.679175 11794 raft_consensus.cc:2749] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:26.679626 11349 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:26.680437 11801 raft_consensus.cc:2804] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:26.680944 11801 raft_consensus.cc:697] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:26.681720 11801 consensus_queue.cc:237] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.684194 11424 ts_tablet_manager.cc:1434] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.069s user 0.010s sys 0.000s
I20260321 02:58:26.685861 11424 tablet_bootstrap.cc:492] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:26.669559 11425 raft_consensus.cc:359] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.693697 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:26.688604 11425 raft_consensus.cc:385] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:26.696393 11425 raft_consensus.cc:740] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:26.697134 11429 tablet_bootstrap.cc:492] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:26.697722 11429 ts_tablet_manager.cc:1403] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.078s user 0.014s sys 0.006s
I20260321 02:58:26.699385 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:26.700130 11306 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:26.700969 11429 raft_consensus.cc:359] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.701851 11429 raft_consensus.cc:385] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:26.702193 11429 raft_consensus.cc:740] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:26.705677 11429 consensus_queue.cc:260] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.703703 11781 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:26.702009 11425 consensus_queue.cc:260] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.706880 11781 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.708760 11425 ts_tablet_manager.cc:1434] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.045s user 0.007s sys 0.000s
I20260321 02:58:26.708822 11781 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:58:26.712095 11349 leader_election.cc:343] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:26.716660 11425 tablet_bootstrap.cc:492] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:26.719287 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:26.719817 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:26.720368 11230 raft_consensus.cc:2468] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:26.721065 11429 ts_tablet_manager.cc:1434] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.023s user 0.005s sys 0.002s
I20260321 02:58:26.721752 11349 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:26.722153 11429 tablet_bootstrap.cc:492] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:26.725198 11781 raft_consensus.cc:2804] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:26.725567 11781 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:26.725890 11781 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
W20260321 02:58:26.725031 11349 leader_election.cc:343] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:26.735548 11306 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:26.738488 11349 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:26.740855 11801 raft_consensus.cc:2804] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:26.741557 11801 raft_consensus.cc:697] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:26.742264 11781 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.742933 11801 consensus_queue.cc:237] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.746528 11424 tablet_bootstrap.cc:654] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:26.759747 11781 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:26.760177 11425 tablet_bootstrap.cc:654] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:26.761047 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:26.761684 11230 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:26.771459 11230 raft_consensus.cc:2468] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:26.761088 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:26.772584 11349 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:58:26.773636 11349 leader_election.cc:343] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:26.774523 11841 raft_consensus.cc:2804] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:26.775110 11841 raft_consensus.cc:697] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:26.775880 11841 consensus_queue.cc:237] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.790815 11429 tablet_bootstrap.cc:654] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:26.793458 11133 catalog_manager.cc:5671] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:26.838080 11788 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:26.838848 11788 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.841150 11788 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:58:26.845340 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 47162.9us
I20260321 02:58:26.849308 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:26.852209 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:26.853739 11230 raft_consensus.cc:2468] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 0.
I20260321 02:58:26.854895 11819 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:26.856767 11424 tablet_bootstrap.cc:492] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
W20260321 02:58:26.856927 11275 leader_election.cc:343] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:26.857493 11424 ts_tablet_manager.cc:1403] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.172s user 0.013s sys 0.000s
I20260321 02:58:26.861480 11424 raft_consensus.cc:359] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.865386 11424 raft_consensus.cc:385] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:26.865860 11424 raft_consensus.cc:740] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:26.868359 11819 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.871747 11306 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:26.886348 11275 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:26.904268 11788 raft_consensus.cc:2804] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:26.904973 11788 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:26.905582 11788 raft_consensus.cc:3060] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:26.871593 11819 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:58:26.999332 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 113084us
I20260321 02:58:27.035727 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:27.013939 11798 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:27.020473 11425 tablet_bootstrap.cc:492] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:27.041651 11425 ts_tablet_manager.cc:1403] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.325s user 0.017s sys 0.020s
I20260321 02:58:27.041493 11798 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.026023 11429 tablet_bootstrap.cc:492] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:27.044296 11429 ts_tablet_manager.cc:1403] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.322s user 0.017s sys 0.018s
I20260321 02:58:27.030220 11788 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.046990 11429 raft_consensus.cc:359] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:26.866820 11424 consensus_queue.cc:260] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.047821 11429 raft_consensus.cc:385] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:27.051834 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:27.052851 11424 ts_tablet_manager.cc:1434] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.195s user 0.009s sys 0.004s
I20260321 02:58:27.056284 11425 raft_consensus.cc:359] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.056895 11425 raft_consensus.cc:385] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:27.049641 11429 raft_consensus.cc:740] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:27.057138 11425 raft_consensus.cc:740] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:27.057792 11425 consensus_queue.cc:260] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.061113 11795 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:27.061596 11795 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.063561 11795 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:27.069802 11801 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:27.070341 11801 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.072362 11801 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:27.076097 11429 consensus_queue.cc:260] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.078462 11429 ts_tablet_manager.cc:1434] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.034s user 0.001s sys 0.004s
W20260321 02:58:27.081764 11275 leader_election.cc:343] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:27.082933 11424 tablet_bootstrap.cc:492] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:27.083889 11429 tablet_bootstrap.cc:492] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:27.084815 11230 raft_consensus.cc:3060] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:27.085623 11798 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:27.088052 11788 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:27.093577 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:27.098033 11305 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:27.098734 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:27.107481 11425 ts_tablet_manager.cc:1434] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.066s user 0.005s sys 0.017s
I20260321 02:58:27.112006 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:27.114655 11236 raft_consensus.cc:3060] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
W20260321 02:58:27.115240 11275 leader_election.cc:343] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:27.116078 11275 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:27.121093 11858 raft_consensus.cc:2749] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:27.124040 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:27.125933 11425 tablet_bootstrap.cc:492] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:27.126070 11308 raft_consensus.cc:2393] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 1: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:27.126334 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:27.127000 11307 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:27.133083 11236 raft_consensus.cc:2468] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:27.135377 11424 tablet_bootstrap.cc:654] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:27.136087 11275 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:27.140106 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:27.140911 11235 raft_consensus.cc:2393] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 1: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:27.141479 11788 raft_consensus.cc:2804] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:27.141986 11788 raft_consensus.cc:697] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:58:27.142825 11788 consensus_queue.cc:237] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.152341 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:27.153257 11236 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
I20260321 02:58:27.154019 11429 tablet_bootstrap.cc:654] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:27.158380 11425 tablet_bootstrap.cc:654] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:27.164649 11349 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:27.162109 11200 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:27.166422 11798 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:27.167091 11798 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:27.167433 11798 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:27.168450 11860 consensus_queue.cc:1048] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:27.178339 11856 consensus_queue.cc:1048] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:27.192018 11134 catalog_manager.cc:5671] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:27.193858 11134 catalog_manager.cc:5671] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:27.196008 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:27.197811 11385 raft_consensus.cc:3060] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:27.208022 11349 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:27.220189 11429 tablet_bootstrap.cc:492] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:27.223718 11424 tablet_bootstrap.cc:492] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:27.224267 11424 ts_tablet_manager.cc:1403] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.142s user 0.024s sys 0.004s
I20260321 02:58:27.224221 11429 ts_tablet_manager.cc:1403] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.141s user 0.018s sys 0.013s
I20260321 02:58:27.245749 11424 raft_consensus.cc:359] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.246577 11424 raft_consensus.cc:385] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:27.246920 11424 raft_consensus.cc:740] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:27.247898 11424 consensus_queue.cc:260] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.251307 11385 raft_consensus.cc:2468] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:27.258479 11424 ts_tablet_manager.cc:1434] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.034s user 0.007s sys 0.001s
I20260321 02:58:27.273252 11429 raft_consensus.cc:359] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.274466 11798 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.275385 11424 tablet_bootstrap.cc:492] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:27.276556 11429 raft_consensus.cc:385] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:27.277568 11429 raft_consensus.cc:740] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:27.281956 11853 raft_consensus.cc:2749] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:27.283450 11130 catalog_manager.cc:5671] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 0 to 1, leader changed from <none> to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
W20260321 02:58:27.329980 11349 consensus_peers.cc:597] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 -> Peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Couldn't send request to peer 0d7efc590a1041a9bd4cd4a971332dfb. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260321 02:58:27.344652 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 49862.7us
I20260321 02:58:27.279588 11429 consensus_queue.cc:260] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.350739 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:27.351310 11235 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
W20260321 02:58:27.352295 11349 consensus_peers.cc:597] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 -> Peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Couldn't send request to peer 83b12b883a7f4129a87dadf5a78f08da. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260321 02:58:27.353166 11349 consensus_peers.cc:597] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 -> Peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Couldn't send request to peer 83b12b883a7f4129a87dadf5a78f08da. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:27.354247 11801 consensus_queue.cc:1048] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:27.363755 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:27.364495 11311 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:27.364543 11235 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:27.339830 11694 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:27.340023 11425 tablet_bootstrap.cc:492] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:27.343144 11798 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:27.363703 11801 consensus_queue.cc:1048] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:27.335676 11424 tablet_bootstrap.cc:654] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:27.381289 11349 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:27.387691 11425 ts_tablet_manager.cc:1403] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.269s user 0.010s sys 0.012s
I20260321 02:58:27.395076 11429 ts_tablet_manager.cc:1434] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.170s user 0.008s sys 0.005s
I20260321 02:58:27.405318 11781 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:27.405912 11781 raft_consensus.cc:697] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:27.411422 11695 consensus_queue.cc:1048] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260321 02:58:27.413550 11429 tablet_bootstrap.cc:492] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:27.396273 11694 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.417578 11781 consensus_queue.cc:237] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.421938 11311 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:27.424374 11859 raft_consensus.cc:2749] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:27.425496 11694 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:27.431696 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:58:27.439678 11349 leader_election.cc:343] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:27.433854 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:58:27.449869 11349 leader_election.cc:343] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:27.450352 11349 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:27.452731 11798 raft_consensus.cc:2749] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:27.456518 11429 tablet_bootstrap.cc:654] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:27.464313 11425 raft_consensus.cc:359] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.465065 11425 raft_consensus.cc:385] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:27.465327 11425 raft_consensus.cc:740] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:27.467653 11425 consensus_queue.cc:260] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.494290 11424 tablet_bootstrap.cc:492] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:27.494823 11424 ts_tablet_manager.cc:1403] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.220s user 0.018s sys 0.007s
I20260321 02:58:27.504417 11130 catalog_manager.cc:5671] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:27.507856 11425 ts_tablet_manager.cc:1434] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.111s user 0.007s sys 0.000s
I20260321 02:58:27.532670 11424 raft_consensus.cc:359] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.533623 11425 tablet_bootstrap.cc:492] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:27.534803 11424 raft_consensus.cc:385] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:27.535115 11424 raft_consensus.cc:740] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
W20260321 02:58:27.547823 11182 auto_rebalancer.cc:247] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260321 02:58:27.536324 11424 consensus_queue.cc:260] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.562996 11788 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:27.575377 11814 consensus_queue.cc:1048] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:27.579663 11424 ts_tablet_manager.cc:1434] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.085s user 0.008s sys 0.000s
I20260321 02:58:27.563634 11788 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.580817 11424 tablet_bootstrap.cc:492] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:27.581919 11788 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:27.587442 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:27.597330 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:27.635844 11425 tablet_bootstrap.cc:654] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:27.636579 11429 tablet_bootstrap.cc:492] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:27.637097 11429 ts_tablet_manager.cc:1403] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.232s user 0.020s sys 0.009s
W20260321 02:58:27.624621 11874 log.cc:927] Time spent T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.068s user 0.000s sys 0.005s
I20260321 02:58:27.639585 11275 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:27.642308 11794 raft_consensus.cc:2749] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:27.663753 11306 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:27.695876 11424 tablet_bootstrap.cc:654] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:27.695279 11777 consensus_queue.cc:1048] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260321 02:58:27.729115 11429 raft_consensus.cc:359] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.729857 11429 raft_consensus.cc:385] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:27.730177 11429 raft_consensus.cc:740] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:27.730680 11429 consensus_queue.cc:260] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.746256 11429 ts_tablet_manager.cc:1434] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.109s user 0.002s sys 0.005s
I20260321 02:58:27.749102 11429 tablet_bootstrap.cc:492] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:27.747920 11425 tablet_bootstrap.cc:492] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:27.752352 11425 ts_tablet_manager.cc:1403] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.219s user 0.012s sys 0.015s
I20260321 02:58:27.762120 11425 raft_consensus.cc:359] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.764353 11425 raft_consensus.cc:385] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:27.766309 11425 raft_consensus.cc:740] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:27.767712 11425 consensus_queue.cc:260] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.771500 11425 ts_tablet_manager.cc:1434] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260321 02:58:27.773869 11429 tablet_bootstrap.cc:654] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:27.777721 11425 tablet_bootstrap.cc:492] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:27.780133 11424 tablet_bootstrap.cc:492] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:27.783322 11424 ts_tablet_manager.cc:1403] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.203s user 0.018s sys 0.008s
I20260321 02:58:27.786286 11424 raft_consensus.cc:359] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.787034 11424 raft_consensus.cc:385] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:27.787298 11424 raft_consensus.cc:740] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:27.788043 11853 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:27.789558 11853 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.793696 11424 consensus_queue.cc:260] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.796123 11424 ts_tablet_manager.cc:1434] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260321 02:58:27.797147 11424 tablet_bootstrap.cc:492] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5: Bootstrap starting.
I20260321 02:58:27.799389 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:27.800977 11856 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:27.823652 11856 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.825628 11856 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:27.804468 11853 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:27.805776 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:27.818076 11429 tablet_bootstrap.cc:492] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:27.822427 11425 tablet_bootstrap.cc:654] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:27.827160 11429 ts_tablet_manager.cc:1403] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.078s user 0.005s sys 0.007s
I20260321 02:58:27.826550 11310 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 0.
W20260321 02:58:27.829437 11349 leader_election.cc:343] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:27.829511 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:27.830637 11424 tablet_bootstrap.cc:654] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:27.830159 11349 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:27.832273 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:27.833081 11383 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:27.836580 11200 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:27.837778 11856 raft_consensus.cc:2804] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:27.838035 11797 raft_consensus.cc:2804] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:27.838289 11856 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:27.838531 11797 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:27.838729 11856 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:27.838955 11797 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
W20260321 02:58:27.842383 11200 leader_election.cc:343] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
I20260321 02:58:27.865947 11856 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.864315 11429 raft_consensus.cc:359] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.873420 11429 raft_consensus.cc:385] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:27.880254 11429 raft_consensus.cc:740] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:27.875721 11798 consensus_queue.cc:1048] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:27.882788 11429 consensus_queue.cc:260] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.884163 11856 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:27.883729 11797 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.886560 11797 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:27.886564 11429 ts_tablet_manager.cc:1434] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.059s user 0.006s sys 0.000s
I20260321 02:58:27.888612 11429 tablet_bootstrap.cc:492] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:27.892068 11424 tablet_bootstrap.cc:492] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5: No bootstrap required, opened a new log
I20260321 02:58:27.892835 11424 ts_tablet_manager.cc:1403] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5: Time spent bootstrapping tablet: real 0.096s user 0.012s sys 0.001s
I20260321 02:58:27.898809 11425 tablet_bootstrap.cc:492] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:27.905665 11425 ts_tablet_manager.cc:1403] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.128s user 0.022s sys 0.010s
I20260321 02:58:27.908383 11424 raft_consensus.cc:359] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.909116 11424 raft_consensus.cc:385] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:27.909431 11424 raft_consensus.cc:740] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Initialized, Role: FOLLOWER
I20260321 02:58:27.901585 11429 tablet_bootstrap.cc:654] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:27.910120 11424 consensus_queue.cc:260] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.916133 11425 raft_consensus.cc:359] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.916893 11425 raft_consensus.cc:385] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:27.918900 11425 raft_consensus.cc:740] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:27.917177 11424 ts_tablet_manager.cc:1434] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260321 02:58:27.919701 11425 consensus_queue.cc:260] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.922312 11425 ts_tablet_manager.cc:1434] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.016s user 0.002s sys 0.003s
I20260321 02:58:27.923470 11425 tablet_bootstrap.cc:492] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:27.929143 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:27.929876 11308 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:27.930213 11425 tablet_bootstrap.cc:654] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:27.940318 11429 tablet_bootstrap.cc:492] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:27.940826 11429 ts_tablet_manager.cc:1403] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.053s user 0.012s sys 0.000s
I20260321 02:58:27.941190 11308 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:27.943113 11429 raft_consensus.cc:359] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.943694 11429 raft_consensus.cc:385] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:27.944556 11349 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:27.943864 11429 raft_consensus.cc:740] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:27.945761 11797 raft_consensus.cc:2804] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:27.946328 11797 raft_consensus.cc:697] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:27.946792 11429 consensus_queue.cc:260] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.949496 11429 ts_tablet_manager.cc:1434] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260321 02:58:27.947156 11797 consensus_queue.cc:237] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.969345 11425 tablet_bootstrap.cc:492] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:27.970232 11425 ts_tablet_manager.cc:1403] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.018s
I20260321 02:58:27.976284 11425 raft_consensus.cc:359] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.977025 11425 raft_consensus.cc:385] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:27.977314 11425 raft_consensus.cc:740] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:27.977635 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:27.978220 11383 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:27.977998 11425 consensus_queue.cc:260] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:27.980587 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:27.980968 11425 ts_tablet_manager.cc:1434] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260321 02:58:27.981207 11235 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:27.981974 11425 tablet_bootstrap.cc:492] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb: Bootstrap starting.
I20260321 02:58:27.983171 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:27.984028 11308 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:27.992237 11425 tablet_bootstrap.cc:654] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:27.993443 11235 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:27.996516 11383 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:28.000469 11308 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:28.011463 11200 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:28.025888 11795 raft_consensus.cc:2804] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:28.067399 11429 tablet_bootstrap.cc:492] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da: Bootstrap starting.
I20260321 02:58:28.114835 11425 tablet_bootstrap.cc:492] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb: No bootstrap required, opened a new log
I20260321 02:58:28.115751 11425 ts_tablet_manager.cc:1403] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent bootstrapping tablet: real 0.134s user 0.012s sys 0.027s
I20260321 02:58:28.119021 11702 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:28.119554 11702 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:28.121731 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 50100.8us
W20260321 02:58:28.128547 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 110565us
I20260321 02:58:28.117214 11795 raft_consensus.cc:697] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:28.147096 11702 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:28.148332 11429 tablet_bootstrap.cc:654] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da: Neither blocks nor log segments found. Creating new log.
I20260321 02:58:28.137076 11425 raft_consensus.cc:359] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:28.156064 11425 raft_consensus.cc:385] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:28.157068 11425 raft_consensus.cc:740] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Initialized, Role: FOLLOWER
I20260321 02:58:28.159049 11425 consensus_queue.cc:260] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:28.161168 11425 ts_tablet_manager.cc:1434] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent starting tablet: real 0.045s user 0.008s sys 0.000s
I20260321 02:58:28.162006 11795 consensus_queue.cc:237] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:28.177867 11814 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:28.178661 11814 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:28.198916 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:58:28.225018 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 85478.5us
I20260321 02:58:28.233620 11855 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:28.234297 11855 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:28.237085 11855 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:28.252194 11235 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:28.260221 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:28.271752 11310 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:28.283164 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:28.286011 11895 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:28.286984 11895 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:28.290761 11895 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:28.549108 11200 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:28.566009 11905 raft_consensus.cc:2804] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:28.566581 11905 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:28.567065 11905 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
W20260321 02:58:28.624977 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 59144.3us
W20260321 02:58:28.636498 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 83761.6us
I20260321 02:58:28.646000 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:58:28.649732 11182 auto_rebalancer.cc:247] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260321 02:58:28.681471 11200 leader_election.cc:343] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Illegal state: must be running to vote when last-logged opid is not known
W20260321 02:58:28.693948 11795 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.013s sys 0.008s
W20260321 02:58:28.697366 11795 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.527s user 0.013s sys 0.009s
I20260321 02:58:28.700500 11905 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:28.707504 11905 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:28.713306 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:28.714309 11306 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:28.724628 11306 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:28.726614 11200 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:28.728119 11699 raft_consensus.cc:2804] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:28.729007 11699 raft_consensus.cc:697] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:28.728922 11905 consensus_queue.cc:1048] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:28.730285 11699 consensus_queue.cc:237] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:28.787374 11814 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:58:28.800861 11429 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.008s sys 0.016s
W20260321 02:58:28.801203 11429 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.601s user 0.008s sys 0.016s
I20260321 02:58:28.803145 11429 tablet_bootstrap.cc:492] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da: No bootstrap required, opened a new log
I20260321 02:58:28.803565 11429 ts_tablet_manager.cc:1403] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da: Time spent bootstrapping tablet: real 0.740s user 0.017s sys 0.023s
I20260321 02:58:28.815045 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:28.815732 11379 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:28.822227 11429 raft_consensus.cc:359] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:28.822988 11429 raft_consensus.cc:385] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260321 02:58:28.823244 11429 raft_consensus.cc:740] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Initialized, Role: FOLLOWER
I20260321 02:58:28.823880 11429 consensus_queue.cc:260] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:28.826359 11429 ts_tablet_manager.cc:1434] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260321 02:58:28.827286 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:28.828882 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:28.827358 11379 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:28.831146 11797 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:28.831318 11376 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 0.
I20260321 02:58:28.831856 11797 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:28.839198 11856 consensus_queue.cc:1048] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:28.840682 11200 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:28.841369 11859 raft_consensus.cc:2804] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260321 02:58:28.841740 11859 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:28.842036 11859 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:28.852618 11859 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:28.854499 11349 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.008s sys 0.039s
W20260321 02:58:28.854851 11349 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.664s user 0.008s sys 0.039s
W20260321 02:58:28.855084 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 664371us
I20260321 02:58:28.868229 11859 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:28.876006 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:28.876740 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:28.876842 11310 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:28.882521 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:28.883184 11379 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:28.892601 11310 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:28.893297 11853 consensus_queue.cc:1048] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:28.893960 11797 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:28.901979 11379 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:28.903614 11200 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:28.904585 11200 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:28.908356 11831 raft_consensus.cc:2804] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260321 02:58:28.909227 11831 raft_consensus.cc:697] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:28.910403 11831 consensus_queue.cc:237] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:28.907167 11130 catalog_manager.cc:5671] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:28.916074 11855 raft_consensus.cc:2749] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:28.921584 11853 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:28.922125 11853 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:28.925649 11853 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:28.926499 11894 consensus_queue.cc:1048] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:28.939116 11307 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 0 FOLLOWER]: Advancing to term 1
I20260321 02:58:28.971345 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:28.974371 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:28.977989 11302 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:28.978652 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:28.982918 11798 consensus_queue.cc:1048] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:28.983179 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:28.972898 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:29.010462 11349 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:29.011173 11853 raft_consensus.cc:2804] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:29.011601 11853 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:29.012063 11853 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:29.012899 11275 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:29.014005 11794 raft_consensus.cc:2749] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260321 02:58:29.020830 11853 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:29.023204 11853 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:29.029350 11349 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:29.030987 11798 raft_consensus.cc:2749] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:29.033073 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:29.033521 11133 catalog_manager.cc:5671] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:29.036156 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:29.036803 11306 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:29.035166 11133 catalog_manager.cc:5671] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:29.060256 11306 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:29.071233 11130 catalog_manager.cc:5671] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:29.076274 11349 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:29.078051 11853 raft_consensus.cc:2804] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:58:29.080710 11853 raft_consensus.cc:697] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:29.081540 11853 consensus_queue.cc:237] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:29.098205 11130 catalog_manager.cc:5671] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 2, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 2 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
W20260321 02:58:29.262859 11332 debug-util.cc:398] Leaking SignalData structure 0x7b080016c500 after lost signal to thread 10071
W20260321 02:58:29.264092 11332 debug-util.cc:398] Leaking SignalData structure 0x7b080012bba0 after lost signal to thread 11168
W20260321 02:58:29.265323 11332 debug-util.cc:398] Leaking SignalData structure 0x7b08002ca4c0 after lost signal to thread 11260
W20260321 02:58:29.266332 11332 debug-util.cc:398] Leaking SignalData structure 0x7b08002c9bc0 after lost signal to thread 11335
W20260321 02:58:29.267388 11332 debug-util.cc:398] Leaking SignalData structure 0x7b08002b7680 after lost signal to thread 11409
I20260321 02:58:29.267644 11307 raft_consensus.cc:1240] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
W20260321 02:58:29.268080 11332 debug-util.cc:398] Leaking SignalData structure 0x7b08002b74a0 after lost signal to thread 11788
W20260321 02:58:29.268502 11332 debug-util.cc:398] Leaking SignalData structure 0x7b08002b75c0 after lost signal to thread 11801
I20260321 02:58:29.269774 11900 consensus_queue.cc:1059] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:29.270871 11831 raft_consensus.cc:3055] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:29.271196 11831 raft_consensus.cc:740] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:29.271976 11831 consensus_queue.cc:260] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:29.277987 11831 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:29.354766 11795 consensus_queue.cc:1048] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:29.370293 11900 consensus_queue.cc:1048] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260321 02:58:29.575451 10068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260321 02:58:29.608383 11932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 02:58:29.614092 11931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260321 02:58:29.621994 11934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260321 02:58:29.622069 10068 server_base.cc:1061] running on GCE node
I20260321 02:58:29.624266 10068 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260321 02:58:29.624707 10068 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260321 02:58:29.625092 10068 hybrid_clock.cc:648] HybridClock initialized: now 1774061909625070 us; error 0 us; skew 500 ppm
I20260321 02:58:29.625962 10068 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260321 02:58:29.641938 10068 webserver.cc:492] Webserver started at http://127.9.213.4:46529/ using document root <none> and password file <none>
I20260321 02:58:29.642552 10068 fs_manager.cc:362] Metadata directory not provided
I20260321 02:58:29.642832 10068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260321 02:58:29.643265 10068 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260321 02:58:29.645098 10068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-3-root/instance:
uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
format_stamp: "Formatted at 2026-03-21 02:58:29 on dist-test-slave-mz2z"
I20260321 02:58:29.650513 10068 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260321 02:58:29.653833 11235 raft_consensus.cc:1275] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260321 02:58:29.654255 11235 pending_rounds.cc:85] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb: Aborting all ops after (but not including) 0
I20260321 02:58:29.654553 11235 pending_rounds.cc:107] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260321 02:58:29.654837 11235 raft_consensus.cc:2889] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:58:29.656327 11857 consensus_queue.cc:1048] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:29.658061 11939 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260321 02:58:29.659888 10068 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.000s sys 0.003s
I20260321 02:58:29.660267 10068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-3-root
uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
format_stamp: "Formatted at 2026-03-21 02:58:29 on dist-test-slave-mz2z"
I20260321 02:58:29.660631 10068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260321 02:58:29.670994 11307 raft_consensus.cc:1275] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260321 02:58:29.678753 11857 consensus_queue.cc:1048] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260321 02:58:29.901576 10068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260321 02:58:29.903167 10068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260321 02:58:29.930297 10068 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260321 02:58:29.934423 10068 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260321 02:58:29.934749 10068 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260321 02:58:29.935081 10068 ts_tablet_manager.cc:616] Registered 0 tablets
I20260321 02:58:29.935298 10068 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260321 02:58:30.062973 10068 rpc_server.cc:307] RPC server started. Bound to: 127.9.213.4:41285
I20260321 02:58:30.063529 12004 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.9.213.4:41285 every 8 connection(s)
I20260321 02:58:30.317636 10068 auto_leader_rebalancer-test.cc:417] add a tserver: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:30.334707 12005 heartbeater.cc:344] Connected to a master server at 127.9.213.62:36307
I20260321 02:58:30.335208 12005 heartbeater.cc:461] Registering TS with master...
I20260321 02:58:30.336318 12005 heartbeater.cc:507] Master 127.9.213.62:36307 requested a full tablet report, sending...
I20260321 02:58:30.338745 11130 ts_manager.cc:194] Registered new tserver with Master: 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:30.340818 11130 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44688
I20260321 02:58:30.998481 11235 consensus_queue.cc:237] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:31.005476 11379 raft_consensus.cc:1275] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:31.007500 12010 consensus_queue.cc:1048] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:31.011134 11306 raft_consensus.cc:1275] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:31.017928 12009 consensus_queue.cc:1048] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:31.052374 12009 raft_consensus.cc:2955] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:31.056659 11379 raft_consensus.cc:2955] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:31.063439 11305 raft_consensus.cc:2955] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:31.092862 11130 catalog_manager.cc:5671] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:31.106874 11377 consensus_queue.cc:237] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
W20260321 02:58:31.137637 11201 consensus_peers.cc:597] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1146889bd4dc49e9a2b67700208bc264. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:31.157851 11306 raft_consensus.cc:1275] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:31.158953 11228 raft_consensus.cc:1275] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:31.161825 11797 consensus_queue.cc:1048] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:31.163378 11945 consensus_queue.cc:1048] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260321 02:58:31.234634 11350 consensus_peers.cc:597] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b270af8b9da943d2b91acbdc0bd603cc. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:31.236260 11945 raft_consensus.cc:2955] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:31.249871 11228 raft_consensus.cc:2955] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:31.262194 11306 raft_consensus.cc:2955] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:31.270762 11228 consensus_queue.cc:237] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:31.287616 11133 catalog_manager.cc:5671] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:31.301251 11377 raft_consensus.cc:1275] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:31.306524 12010 consensus_queue.cc:1048] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
W20260321 02:58:31.320075 11201 consensus_peers.cc:597] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d5b2547bde048c88e963f650d905f76. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:31.321353 11306 raft_consensus.cc:1275] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:31.321113 12009 raft_consensus.cc:2955] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:31.322723 12017 consensus_queue.cc:1048] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:31.323110 11379 raft_consensus.cc:2955] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:31.344156 12005 heartbeater.cc:499] Master 127.9.213.62:36307 was elected leader, sending a full tablet report...
I20260321 02:58:31.346050 11306 raft_consensus.cc:2955] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:31.347609 11133 catalog_manager.cc:5671] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:31.375484 11379 consensus_queue.cc:237] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
W20260321 02:58:31.395807 11350 consensus_peers.cc:597] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62b43ed6f6404b7aa8463e168f98232a. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:31.396737 11306 raft_consensus.cc:1275] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:31.398466 11945 consensus_queue.cc:1048] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:31.400303 11228 raft_consensus.cc:1275] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:31.412264 11945 consensus_queue.cc:1048] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:31.424872 11798 raft_consensus.cc:2955] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:31.430001 11302 raft_consensus.cc:2955] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:31.439399 11228 raft_consensus.cc:2955] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:31.463653 11133 catalog_manager.cc:5671] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:31.575153 12038 ts_tablet_manager.cc:933] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717)
I20260321 02:58:31.585001 12038 tablet_copy_client.cc:323] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.1:36717
I20260321 02:58:31.626477 11246 tablet_copy_service.cc:140] P 0d7efc590a1041a9bd4cd4a971332dfb: Received BeginTabletCopySession request for tablet 1146889bd4dc49e9a2b67700208bc264 from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:59202)
I20260321 02:58:31.627187 11246 tablet_copy_service.cc:161] P 0d7efc590a1041a9bd4cd4a971332dfb: Beginning new tablet copy session on tablet 1146889bd4dc49e9a2b67700208bc264 from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:59202: session id = 3d8d3176d6ec407ea2a965556c2e2c56-1146889bd4dc49e9a2b67700208bc264
I20260321 02:58:31.637300 11246 tablet_copy_source_session.cc:215] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:58:31.642766 12038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1146889bd4dc49e9a2b67700208bc264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:31.661998 12038 tablet_copy_client.cc:806] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
I20260321 02:58:31.662714 12038 tablet_copy_client.cc:670] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:58:31.676476 12038 tablet_copy_client.cc:538] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 02:58:31.688027 12038 tablet_bootstrap.cc:492] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap starting.
I20260321 02:58:31.727303 12038 tablet_bootstrap.cc:492] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 02:58:31.728513 12038 tablet_bootstrap.cc:492] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap complete.
I20260321 02:58:31.729395 12038 ts_tablet_manager.cc:1403] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent bootstrapping tablet: real 0.042s user 0.017s sys 0.018s
I20260321 02:58:31.733117 12044 ts_tablet_manager.cc:933] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:31.733482 12038 raft_consensus.cc:359] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:31.734541 12038 raft_consensus.cc:740] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Initialized, Role: LEARNER
I20260321 02:58:31.736330 12038 consensus_queue.cc:260] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:31.738386 12044 tablet_copy_client.cc:323] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.3:43533
I20260321 02:58:31.745326 12038 ts_tablet_manager.cc:1434] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260321 02:58:31.756016 11246 tablet_copy_service.cc:342] P 0d7efc590a1041a9bd4cd4a971332dfb: Request end of tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-1146889bd4dc49e9a2b67700208bc264 received from {username='slave'} at 127.0.0.1:59202
I20260321 02:58:31.756600 11246 tablet_copy_service.cc:434] P 0d7efc590a1041a9bd4cd4a971332dfb: ending tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-1146889bd4dc49e9a2b67700208bc264 on tablet 1146889bd4dc49e9a2b67700208bc264 with peer 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:31.788163 12038 ts_tablet_manager.cc:933] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:31.797731 12038 tablet_copy_client.cc:323] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.3:43533
I20260321 02:58:31.804231 12049 ts_tablet_manager.cc:933] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717)
I20260321 02:58:31.808241 12049 tablet_copy_client.cc:323] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.1:36717
I20260321 02:58:31.809954 11246 tablet_copy_service.cc:140] P 0d7efc590a1041a9bd4cd4a971332dfb: Received BeginTabletCopySession request for tablet 7d5b2547bde048c88e963f650d905f76 from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:59202)
I20260321 02:58:31.810547 11246 tablet_copy_service.cc:161] P 0d7efc590a1041a9bd4cd4a971332dfb: Beginning new tablet copy session on tablet 7d5b2547bde048c88e963f650d905f76 from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:59202: session id = 3d8d3176d6ec407ea2a965556c2e2c56-7d5b2547bde048c88e963f650d905f76
I20260321 02:58:31.819720 11394 tablet_copy_service.cc:140] P fef7f8cc55b043039131bd9b2229f0a5: Received BeginTabletCopySession request for tablet 62b43ed6f6404b7aa8463e168f98232a from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:45634)
I20260321 02:58:31.819828 11395 tablet_copy_service.cc:140] P fef7f8cc55b043039131bd9b2229f0a5: Received BeginTabletCopySession request for tablet b270af8b9da943d2b91acbdc0bd603cc from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:45634)
I20260321 02:58:31.820369 11394 tablet_copy_service.cc:161] P fef7f8cc55b043039131bd9b2229f0a5: Beginning new tablet copy session on tablet 62b43ed6f6404b7aa8463e168f98232a from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:45634: session id = 3d8d3176d6ec407ea2a965556c2e2c56-62b43ed6f6404b7aa8463e168f98232a
I20260321 02:58:31.821214 11395 tablet_copy_service.cc:161] P fef7f8cc55b043039131bd9b2229f0a5: Beginning new tablet copy session on tablet b270af8b9da943d2b91acbdc0bd603cc from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:45634: session id = 3d8d3176d6ec407ea2a965556c2e2c56-b270af8b9da943d2b91acbdc0bd603cc
I20260321 02:58:31.822757 11246 tablet_copy_source_session.cc:215] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:58:31.829435 11394 tablet_copy_source_session.cc:215] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:58:31.832353 12049 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d5b2547bde048c88e963f650d905f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:31.836733 12038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62b43ed6f6404b7aa8463e168f98232a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:31.854578 12038 tablet_copy_client.cc:806] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
I20260321 02:58:31.855230 12038 tablet_copy_client.cc:670] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:58:31.859793 12038 tablet_copy_client.cc:538] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 02:58:31.862905 12049 tablet_copy_client.cc:806] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
I20260321 02:58:31.863572 12049 tablet_copy_client.cc:670] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:58:31.864593 11395 tablet_copy_source_session.cc:215] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:58:31.884112 12044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b270af8b9da943d2b91acbdc0bd603cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:31.885963 12049 tablet_copy_client.cc:538] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 02:58:31.899204 12049 tablet_bootstrap.cc:492] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap starting.
I20260321 02:58:31.931936 12049 tablet_bootstrap.cc:492] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 02:58:31.933106 12049 tablet_bootstrap.cc:492] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap complete.
I20260321 02:58:31.933866 12049 ts_tablet_manager.cc:1403] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent bootstrapping tablet: real 0.035s user 0.025s sys 0.008s
I20260321 02:58:31.934725 12038 tablet_bootstrap.cc:492] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap starting.
I20260321 02:58:31.937111 12049 raft_consensus.cc:359] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:31.938112 12049 raft_consensus.cc:740] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Initialized, Role: LEARNER
I20260321 02:58:31.938844 12049 consensus_queue.cc:260] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:31.948660 12049 ts_tablet_manager.cc:1434] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260321 02:58:32.051265 12044 tablet_copy_client.cc:806] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
I20260321 02:58:32.053359 12044 tablet_copy_client.cc:670] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:58:32.080421 11980 raft_consensus.cc:1217] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260321 02:58:32.075677 12044 tablet_copy_client.cc:538] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 02:58:32.102895 11246 tablet_copy_service.cc:342] P 0d7efc590a1041a9bd4cd4a971332dfb: Request end of tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-7d5b2547bde048c88e963f650d905f76 received from {username='slave'} at 127.0.0.1:59202
I20260321 02:58:32.103416 11246 tablet_copy_service.cc:434] P 0d7efc590a1041a9bd4cd4a971332dfb: ending tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-7d5b2547bde048c88e963f650d905f76 on tablet 7d5b2547bde048c88e963f650d905f76 with peer 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:32.203451 12044 tablet_bootstrap.cc:492] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap starting.
W20260321 02:58:32.208932 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 105294us
I20260321 02:58:32.240345 11980 raft_consensus.cc:1217] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20260321 02:58:32.273398 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 84969us
I20260321 02:58:32.297680 12038 tablet_bootstrap.cc:492] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 02:58:32.298903 12038 tablet_bootstrap.cc:492] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap complete.
I20260321 02:58:32.299706 12038 ts_tablet_manager.cc:1403] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent bootstrapping tablet: real 0.365s user 0.134s sys 0.174s
I20260321 02:58:32.304363 12038 raft_consensus.cc:359] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:32.305341 12038 raft_consensus.cc:740] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Initialized, Role: LEARNER
I20260321 02:58:32.306056 12038 consensus_queue.cc:260] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:32.309103 12038 ts_tablet_manager.cc:1434] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent starting tablet: real 0.009s user 0.002s sys 0.005s
I20260321 02:58:32.311630 11395 tablet_copy_service.cc:342] P fef7f8cc55b043039131bd9b2229f0a5: Request end of tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-62b43ed6f6404b7aa8463e168f98232a received from {username='slave'} at 127.0.0.1:45634
I20260321 02:58:32.312132 11395 tablet_copy_service.cc:434] P fef7f8cc55b043039131bd9b2229f0a5: ending tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-62b43ed6f6404b7aa8463e168f98232a on tablet 62b43ed6f6404b7aa8463e168f98232a with peer 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:32.345088 12044 tablet_bootstrap.cc:492] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 02:58:32.346282 12044 tablet_bootstrap.cc:492] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap complete.
I20260321 02:58:32.347047 12044 ts_tablet_manager.cc:1403] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent bootstrapping tablet: real 0.155s user 0.016s sys 0.013s
I20260321 02:58:32.350668 12044 raft_consensus.cc:359] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:32.354367 12044 raft_consensus.cc:740] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Initialized, Role: LEARNER
I20260321 02:58:32.355588 12044 consensus_queue.cc:260] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:32.366492 12044 ts_tablet_manager.cc:1434] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260321 02:58:32.380767 11395 tablet_copy_service.cc:342] P fef7f8cc55b043039131bd9b2229f0a5: Request end of tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-b270af8b9da943d2b91acbdc0bd603cc received from {username='slave'} at 127.0.0.1:45634
I20260321 02:58:32.381253 11395 tablet_copy_service.cc:434] P fef7f8cc55b043039131bd9b2229f0a5: ending tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-b270af8b9da943d2b91acbdc0bd603cc on tablet b270af8b9da943d2b91acbdc0bd603cc with peer 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:32.387017 11980 raft_consensus.cc:1217] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260321 02:58:32.536798 12035 raft_consensus.cc:1064] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb: attempting to promote NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 to VOTER
I20260321 02:58:32.540406 12035 consensus_queue.cc:237] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:32.556564 11980 raft_consensus.cc:1217] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260321 02:58:32.565177 11980 raft_consensus.cc:1275] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:32.565979 11306 raft_consensus.cc:1275] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:32.567260 11381 raft_consensus.cc:1275] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:32.573767 12056 consensus_queue.cc:1048] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:32.574786 12053 consensus_queue.cc:1048] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:32.577798 12053 consensus_queue.cc:1048] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:32.604878 12053 raft_consensus.cc:2955] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:32.617995 11302 raft_consensus.cc:2955] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:32.626085 11381 raft_consensus.cc:2955] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:32.642058 11980 raft_consensus.cc:2955] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:32.635874 11133 catalog_manager.cc:5671] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:58:32.705874 11235 consensus_queue.cc:237] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:32.713910 11980 raft_consensus.cc:1275] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:32.715659 12053 consensus_queue.cc:1048] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:32.722981 11302 raft_consensus.cc:1275] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:32.726866 12017 raft_consensus.cc:2955] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:32.729637 11980 raft_consensus.cc:2955] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:32.739100 12035 consensus_queue.cc:1048] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:32.745657 11119 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 1146889bd4dc49e9a2b67700208bc264 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260321 02:58:32.757444 11132 catalog_manager.cc:5671] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:58:32.762353 11306 raft_consensus.cc:2955] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:32.779515 12056 raft_consensus.cc:1064] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb: attempting to promote NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 to VOTER
I20260321 02:58:32.782327 12056 consensus_queue.cc:237] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:32.800709 11366 tablet_service.cc:1558] Processing DeleteTablet for tablet 1146889bd4dc49e9a2b67700208bc264 with delete_type TABLET_DATA_TOMBSTONED (TS fef7f8cc55b043039131bd9b2229f0a5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40932
I20260321 02:58:32.813947 12070 tablet_replica.cc:333] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5: stopping tablet replica
I20260321 02:58:32.819662 12057 raft_consensus.cc:1064] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5: attempting to promote NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 to VOTER
I20260321 02:58:32.822448 12057 consensus_queue.cc:237] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:32.824276 12070 raft_consensus.cc:2243] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:32.825193 12070 raft_consensus.cc:2272] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:32.830132 12070 ts_tablet_manager.cc:1916] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 02:58:32.838943 11381 raft_consensus.cc:1275] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:32.843590 11980 raft_consensus.cc:1275] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:32.846130 12017 consensus_queue.cc:1048] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:32.850541 12017 consensus_queue.cc:1048] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:32.857371 11306 raft_consensus.cc:1275] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:32.865370 12017 consensus_queue.cc:1048] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:32.872951 12070 ts_tablet_manager.cc:1929] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260321 02:58:32.873535 12070 log.cc:1199] T 1146889bd4dc49e9a2b67700208bc264 P fef7f8cc55b043039131bd9b2229f0a5: Deleting WAL directory at /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-2-root/wals/1146889bd4dc49e9a2b67700208bc264
I20260321 02:58:32.876101 11119 catalog_manager.cc:5002] TS fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): tablet 1146889bd4dc49e9a2b67700208bc264 (table test-workload [id=07ae567e73b94faa9ae558018daa9713]) successfully deleted
I20260321 02:58:32.915212 11980 raft_consensus.cc:1275] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:32.916776 12025 consensus_queue.cc:1048] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:32.922318 11230 raft_consensus.cc:1275] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:32.923866 11302 raft_consensus.cc:1275] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:32.926297 12058 consensus_queue.cc:1048] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:32.929751 12057 consensus_queue.cc:1048] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:32.929445 12017 raft_consensus.cc:2955] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:32.931337 11306 raft_consensus.cc:2955] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:32.984476 11980 raft_consensus.cc:2955] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:32.997170 11377 raft_consensus.cc:2955] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.011868 11130 catalog_manager.cc:5671] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.026228 11798 raft_consensus.cc:2955] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.028806 11306 raft_consensus.cc:2955] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.033951 11230 raft_consensus.cc:2955] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.059263 11980 raft_consensus.cc:2955] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.085889 11131 catalog_manager.cc:5671] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.126243 12025 raft_consensus.cc:1064] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5: attempting to promote NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 to VOTER
I20260321 02:58:33.129033 12025 consensus_queue.cc:237] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:33.138270 11236 consensus_queue.cc:237] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:33.145258 11980 raft_consensus.cc:1275] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:33.147244 12058 consensus_queue.cc:1048] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:33.151614 11381 consensus_queue.cc:237] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:33.155507 11306 raft_consensus.cc:1275] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:33.160559 12057 consensus_queue.cc:1048] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:33.163820 11979 raft_consensus.cc:1275] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:33.166500 11979 raft_consensus.cc:1275] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:33.168459 11798 consensus_queue.cc:1048] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:33.171375 11230 raft_consensus.cc:1275] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:33.171375 12056 consensus_queue.cc:1048] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260321 02:58:33.183341 12017 raft_consensus.cc:2955] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.190344 11235 raft_consensus.cc:1275] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:33.190932 11978 raft_consensus.cc:2955] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.194003 11381 raft_consensus.cc:1275] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:33.200533 12055 raft_consensus.cc:2955] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.205235 12056 consensus_queue.cc:1048] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:33.210608 11979 raft_consensus.cc:2955] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.204442 12058 consensus_queue.cc:1048] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20260321 02:58:33.220330 11376 raft_consensus.cc:2955] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.241734 12057 consensus_queue.cc:1048] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:33.252241 11119 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 62b43ed6f6404b7aa8463e168f98232a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260321 02:58:33.254987 12057 raft_consensus.cc:2955] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
W20260321 02:58:33.261235 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 65866.6us
I20260321 02:58:33.265197 11977 raft_consensus.cc:2955] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.277639 11230 raft_consensus.cc:2955] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.296856 11119 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 7d5b2547bde048c88e963f650d905f76 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260321 02:58:33.296995 11131 catalog_manager.cc:5671] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:58:33.299396 11133 catalog_manager.cc:5671] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.331715 11299 tablet_service.cc:1558] Processing DeleteTablet for tablet 62b43ed6f6404b7aa8463e168f98232a with delete_type TABLET_DATA_TOMBSTONED (TS 83b12b883a7f4129a87dadf5a78f08da not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56610
I20260321 02:58:33.333581 11134 catalog_manager.cc:5671] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.342933 11301 tablet_service.cc:1558] Processing DeleteTablet for tablet 7d5b2547bde048c88e963f650d905f76 with delete_type TABLET_DATA_TOMBSTONED (TS 83b12b883a7f4129a87dadf5a78f08da not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56610
I20260321 02:58:33.352283 11302 raft_consensus.cc:2955] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.362191 11381 consensus_queue.cc:237] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:33.386209 12085 tablet_replica.cc:333] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da: stopping tablet replica
I20260321 02:58:33.389973 12085 raft_consensus.cc:2243] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:33.392895 12085 raft_consensus.cc:2272] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:33.402308 11977 raft_consensus.cc:1275] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:33.405134 12055 consensus_queue.cc:1048] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:33.421650 12085 ts_tablet_manager.cc:1916] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 02:58:33.421188 11979 raft_consensus.cc:2955] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.424706 12058 raft_consensus.cc:2955] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.443073 12085 ts_tablet_manager.cc:1929] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260321 02:58:33.444043 12085 log.cc:1199] T 62b43ed6f6404b7aa8463e168f98232a P 83b12b883a7f4129a87dadf5a78f08da: Deleting WAL directory at /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-1-root/wals/62b43ed6f6404b7aa8463e168f98232a
I20260321 02:58:33.446305 12085 tablet_replica.cc:333] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da: stopping tablet replica
I20260321 02:58:33.447578 12085 raft_consensus.cc:2243] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:33.448472 12085 raft_consensus.cc:2272] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:33.453176 12085 ts_tablet_manager.cc:1916] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 02:58:33.453845 11119 catalog_manager.cc:5002] TS 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): tablet 62b43ed6f6404b7aa8463e168f98232a (table test-workload [id=07ae567e73b94faa9ae558018daa9713]) successfully deleted
I20260321 02:58:33.454548 11308 raft_consensus.cc:1275] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:33.459625 11119 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet b270af8b9da943d2b91acbdc0bd603cc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260321 02:58:33.459079 11133 catalog_manager.cc:5671] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.463708 12055 consensus_queue.cc:1048] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:33.472385 11217 tablet_service.cc:1558] Processing DeleteTablet for tablet b270af8b9da943d2b91acbdc0bd603cc with delete_type TABLET_DATA_TOMBSTONED (TS 0d7efc590a1041a9bd4cd4a971332dfb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58196
I20260321 02:58:33.475642 11307 raft_consensus.cc:2955] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:33.501749 12087 tablet_replica.cc:333] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb: stopping tablet replica
I20260321 02:58:33.507980 12087 raft_consensus.cc:2243] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:33.510710 12087 pending_rounds.cc:70] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb: Trying to abort 1 pending ops.
I20260321 02:58:33.511057 12087 pending_rounds.cc:77] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7266557596169555968 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } } }
I20260321 02:58:33.514117 12085 ts_tablet_manager.cc:1929] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260321 02:58:33.514132 12087 raft_consensus.cc:2917] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20260321 02:58:33.514780 12085 log.cc:1199] T 7d5b2547bde048c88e963f650d905f76 P 83b12b883a7f4129a87dadf5a78f08da: Deleting WAL directory at /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-1-root/wals/7d5b2547bde048c88e963f650d905f76
I20260321 02:58:33.518148 12087 raft_consensus.cc:2889] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260321 02:58:33.518788 12087 raft_consensus.cc:2272] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:33.520095 11119 catalog_manager.cc:5002] TS 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): tablet 7d5b2547bde048c88e963f650d905f76 (table test-workload [id=07ae567e73b94faa9ae558018daa9713]) successfully deleted
I20260321 02:58:33.522976 12087 ts_tablet_manager.cc:1916] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 02:58:33.540563 12087 ts_tablet_manager.cc:1929] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260321 02:58:33.540908 12087 log.cc:1199] T b270af8b9da943d2b91acbdc0bd603cc P 0d7efc590a1041a9bd4cd4a971332dfb: Deleting WAL directory at /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-0-root/wals/b270af8b9da943d2b91acbdc0bd603cc
I20260321 02:58:33.542268 11119 catalog_manager.cc:5002] TS 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): tablet b270af8b9da943d2b91acbdc0bd603cc (table test-workload [id=07ae567e73b94faa9ae558018daa9713]) successfully deleted
I20260321 02:58:34.648659 11307 consensus_queue.cc:237] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:34.659412 11236 raft_consensus.cc:1275] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:34.661934 12082 consensus_queue.cc:1048] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:34.662130 11376 raft_consensus.cc:1275] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:34.673277 12088 consensus_queue.cc:1048] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:34.683892 11880 raft_consensus.cc:2955] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:34.685834 11232 raft_consensus.cc:2955] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
W20260321 02:58:34.699126 11276 consensus_peers.cc:597] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b8d0e26d25e4888bc95c0956520bdd1. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:34.704372 11236 consensus_queue.cc:237] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:34.721508 11133 catalog_manager.cc:5671] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260321 02:58:34.737751 11201 consensus_peers.cc:597] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4337508daeb4d598c4f6a0b4314878d. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:34.743299 11376 raft_consensus.cc:1275] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:34.742826 11384 raft_consensus.cc:2955] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:34.746373 11304 raft_consensus.cc:1275] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:34.749543 12089 consensus_queue.cc:1048] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:34.776263 12089 consensus_queue.cc:1048] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:34.788316 12035 raft_consensus.cc:2955] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:34.791220 11304 raft_consensus.cc:2955] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:34.808933 11307 consensus_queue.cc:237] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:34.821245 11133 catalog_manager.cc:5671] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:34.823027 11376 raft_consensus.cc:2955] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
W20260321 02:58:34.833475 11276 consensus_peers.cc:597] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a42977f759be4144a5862197a77e395d. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:34.833616 11384 raft_consensus.cc:1275] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:34.836015 12082 consensus_queue.cc:1048] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:34.840626 11236 raft_consensus.cc:1275] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:34.847149 11880 consensus_queue.cc:1048] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:34.872689 12082 raft_consensus.cc:2955] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:34.876566 11236 raft_consensus.cc:2955] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:34.874713 11376 raft_consensus.cc:2955] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:34.903129 11134 catalog_manager.cc:5671] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:34.910252 11230 consensus_queue.cc:237] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
W20260321 02:58:34.930306 11201 consensus_peers.cc:597] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 006e9dd81c0c4368bed549c97128f4ea. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:34.930379 11376 raft_consensus.cc:1275] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:34.931980 11307 raft_consensus.cc:1275] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:34.932119 12089 consensus_queue.cc:1048] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:34.933528 12105 consensus_queue.cc:1048] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:34.946946 12089 raft_consensus.cc:2955] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:34.952913 11307 raft_consensus.cc:2955] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:34.954248 11376 raft_consensus.cc:2955] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:34.976123 11134 catalog_manager.cc:5671] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:35.050637 12113 ts_tablet_manager.cc:933] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:35.054407 12113 tablet_copy_client.cc:323] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.2:34267
I20260321 02:58:35.106112 11321 tablet_copy_service.cc:140] P 83b12b883a7f4129a87dadf5a78f08da: Received BeginTabletCopySession request for tablet 1b8d0e26d25e4888bc95c0956520bdd1 from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:49786)
I20260321 02:58:35.106809 11321 tablet_copy_service.cc:161] P 83b12b883a7f4129a87dadf5a78f08da: Beginning new tablet copy session on tablet 1b8d0e26d25e4888bc95c0956520bdd1 from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:49786: session id = 3d8d3176d6ec407ea2a965556c2e2c56-1b8d0e26d25e4888bc95c0956520bdd1
I20260321 02:58:35.115875 11321 tablet_copy_source_session.cc:215] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:58:35.121176 12113 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b8d0e26d25e4888bc95c0956520bdd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:35.162707 12113 tablet_copy_client.cc:806] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
I20260321 02:58:35.163298 12113 tablet_copy_client.cc:670] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:58:35.169955 12113 tablet_copy_client.cc:538] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 02:58:35.172305 12116 ts_tablet_manager.cc:933] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717)
I20260321 02:58:35.175490 12116 tablet_copy_client.cc:323] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.1:36717
I20260321 02:58:35.177292 11246 tablet_copy_service.cc:140] P 0d7efc590a1041a9bd4cd4a971332dfb: Received BeginTabletCopySession request for tablet e4337508daeb4d598c4f6a0b4314878d from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:59202)
I20260321 02:58:35.178041 11246 tablet_copy_service.cc:161] P 0d7efc590a1041a9bd4cd4a971332dfb: Beginning new tablet copy session on tablet e4337508daeb4d598c4f6a0b4314878d from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:59202: session id = 3d8d3176d6ec407ea2a965556c2e2c56-e4337508daeb4d598c4f6a0b4314878d
I20260321 02:58:35.185832 12113 tablet_bootstrap.cc:492] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap starting.
I20260321 02:58:35.195719 11246 tablet_copy_source_session.cc:215] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:58:35.200842 12116 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4337508daeb4d598c4f6a0b4314878d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:35.226208 12116 tablet_copy_client.cc:806] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
I20260321 02:58:35.227061 12116 tablet_copy_client.cc:670] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:58:35.234356 12116 tablet_copy_client.cc:538] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 02:58:35.241338 12113 tablet_bootstrap.cc:492] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 02:58:35.242429 12113 tablet_bootstrap.cc:492] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap complete.
I20260321 02:58:35.243199 12113 ts_tablet_manager.cc:1403] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent bootstrapping tablet: real 0.058s user 0.012s sys 0.017s
I20260321 02:58:35.244994 12116 tablet_bootstrap.cc:492] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap starting.
I20260321 02:58:35.246587 12113 raft_consensus.cc:359] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:35.247447 12113 raft_consensus.cc:740] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Initialized, Role: LEARNER
I20260321 02:58:35.248216 12113 consensus_queue.cc:260] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:35.258114 12113 ts_tablet_manager.cc:1434] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260321 02:58:35.274307 11321 tablet_copy_service.cc:342] P 83b12b883a7f4129a87dadf5a78f08da: Request end of tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-1b8d0e26d25e4888bc95c0956520bdd1 received from {username='slave'} at 127.0.0.1:49786
I20260321 02:58:35.275141 11321 tablet_copy_service.cc:434] P 83b12b883a7f4129a87dadf5a78f08da: ending tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-1b8d0e26d25e4888bc95c0956520bdd1 on tablet 1b8d0e26d25e4888bc95c0956520bdd1 with peer 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:35.286574 12116 tablet_bootstrap.cc:492] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 02:58:35.287568 12116 tablet_bootstrap.cc:492] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap complete.
I20260321 02:58:35.288278 12116 ts_tablet_manager.cc:1403] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent bootstrapping tablet: real 0.044s user 0.020s sys 0.011s
I20260321 02:58:35.290944 12116 raft_consensus.cc:359] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:35.291563 12116 raft_consensus.cc:740] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Initialized, Role: LEARNER
I20260321 02:58:35.292225 12116 consensus_queue.cc:260] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:35.297806 12116 ts_tablet_manager.cc:1434] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260321 02:58:35.299642 11246 tablet_copy_service.cc:342] P 0d7efc590a1041a9bd4cd4a971332dfb: Request end of tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-e4337508daeb4d598c4f6a0b4314878d received from {username='slave'} at 127.0.0.1:59202
I20260321 02:58:35.300261 11246 tablet_copy_service.cc:434] P 0d7efc590a1041a9bd4cd4a971332dfb: ending tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-e4337508daeb4d598c4f6a0b4314878d on tablet e4337508daeb4d598c4f6a0b4314878d with peer 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:35.391659 12116 ts_tablet_manager.cc:933] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:35.405179 12116 tablet_copy_client.cc:323] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.2:34267
I20260321 02:58:35.407867 11321 tablet_copy_service.cc:140] P 83b12b883a7f4129a87dadf5a78f08da: Received BeginTabletCopySession request for tablet a42977f759be4144a5862197a77e395d from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:49786)
I20260321 02:58:35.409649 11321 tablet_copy_service.cc:161] P 83b12b883a7f4129a87dadf5a78f08da: Beginning new tablet copy session on tablet a42977f759be4144a5862197a77e395d from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:49786: session id = 3d8d3176d6ec407ea2a965556c2e2c56-a42977f759be4144a5862197a77e395d
I20260321 02:58:35.420046 11321 tablet_copy_source_session.cc:215] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:58:35.423328 12116 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a42977f759be4144a5862197a77e395d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:35.444586 12116 tablet_copy_client.cc:806] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
I20260321 02:58:35.445403 12116 tablet_copy_client.cc:670] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:58:35.455386 11979 raft_consensus.cc:1217] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260321 02:58:35.463653 12116 tablet_copy_client.cc:538] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 02:58:35.480855 12116 tablet_bootstrap.cc:492] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap starting.
I20260321 02:58:35.497833 12113 ts_tablet_manager.cc:933] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717)
I20260321 02:58:35.514500 12113 tablet_copy_client.cc:323] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.1:36717
I20260321 02:58:35.516136 11246 tablet_copy_service.cc:140] P 0d7efc590a1041a9bd4cd4a971332dfb: Received BeginTabletCopySession request for tablet 006e9dd81c0c4368bed549c97128f4ea from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:59202)
I20260321 02:58:35.516893 11246 tablet_copy_service.cc:161] P 0d7efc590a1041a9bd4cd4a971332dfb: Beginning new tablet copy session on tablet 006e9dd81c0c4368bed549c97128f4ea from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:59202: session id = 3d8d3176d6ec407ea2a965556c2e2c56-006e9dd81c0c4368bed549c97128f4ea
I20260321 02:58:35.528373 12116 tablet_bootstrap.cc:492] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 02:58:35.528687 11246 tablet_copy_source_session.cc:215] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:58:35.529963 12116 tablet_bootstrap.cc:492] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap complete.
I20260321 02:58:35.530701 12116 ts_tablet_manager.cc:1403] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent bootstrapping tablet: real 0.050s user 0.026s sys 0.003s
I20260321 02:58:35.534682 12113 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 006e9dd81c0c4368bed549c97128f4ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:35.535663 12116 raft_consensus.cc:359] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:35.536830 12116 raft_consensus.cc:740] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Initialized, Role: LEARNER
I20260321 02:58:35.537592 12116 consensus_queue.cc:260] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:35.541816 12116 ts_tablet_manager.cc:1434] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260321 02:58:35.546307 11321 tablet_copy_service.cc:342] P 83b12b883a7f4129a87dadf5a78f08da: Request end of tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-a42977f759be4144a5862197a77e395d received from {username='slave'} at 127.0.0.1:49786
I20260321 02:58:35.546897 11321 tablet_copy_service.cc:434] P 83b12b883a7f4129a87dadf5a78f08da: ending tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-a42977f759be4144a5862197a77e395d on tablet a42977f759be4144a5862197a77e395d with peer 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:35.586352 12113 tablet_copy_client.cc:806] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
I20260321 02:58:35.587067 12113 tablet_copy_client.cc:670] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:58:35.594679 12113 tablet_copy_client.cc:538] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 02:58:35.603600 12113 tablet_bootstrap.cc:492] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap starting.
I20260321 02:58:35.656384 12113 tablet_bootstrap.cc:492] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 02:58:35.657502 12113 tablet_bootstrap.cc:492] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap complete.
I20260321 02:58:35.658346 12113 ts_tablet_manager.cc:1403] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent bootstrapping tablet: real 0.055s user 0.015s sys 0.014s
I20260321 02:58:35.660833 12113 raft_consensus.cc:359] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:35.662308 12113 raft_consensus.cc:740] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Initialized, Role: LEARNER
I20260321 02:58:35.663015 12113 consensus_queue.cc:260] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:35.670455 12113 ts_tablet_manager.cc:1434] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260321 02:58:35.679807 11246 tablet_copy_service.cc:342] P 0d7efc590a1041a9bd4cd4a971332dfb: Request end of tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-006e9dd81c0c4368bed549c97128f4ea received from {username='slave'} at 127.0.0.1:59202
I20260321 02:58:35.680301 11246 tablet_copy_service.cc:434] P 0d7efc590a1041a9bd4cd4a971332dfb: ending tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-006e9dd81c0c4368bed549c97128f4ea on tablet 006e9dd81c0c4368bed549c97128f4ea with peer 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:35.725226 11979 raft_consensus.cc:1217] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260321 02:58:35.818444 11979 raft_consensus.cc:1217] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260321 02:58:35.914103 12125 raft_consensus.cc:1064] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da: attempting to promote NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 to VOTER
I20260321 02:58:35.917549 12125 consensus_queue.cc:237] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:35.936528 11979 raft_consensus.cc:1275] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:35.940743 11381 raft_consensus.cc:1275] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:35.941598 11236 raft_consensus.cc:1275] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:35.942745 12082 consensus_queue.cc:1048] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:35.946964 12082 consensus_queue.cc:1048] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:35.964741 12082 consensus_queue.cc:1048] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:35.989521 11384 raft_consensus.cc:2955] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:35.996766 11232 raft_consensus.cc:2955] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:35.987154 12126 raft_consensus.cc:2955] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.016567 11131 catalog_manager.cc:5671] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.036196 11979 raft_consensus.cc:2955] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.073128 11979 raft_consensus.cc:1217] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260321 02:58:36.093482 11304 consensus_queue.cc:237] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:36.099870 11979 raft_consensus.cc:1275] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:36.101576 11381 raft_consensus.cc:1275] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:36.105376 12132 consensus_queue.cc:1048] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:36.106703 12125 consensus_queue.cc:1048] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:36.117297 12125 raft_consensus.cc:2955] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.119341 11384 raft_consensus.cc:2955] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.129820 11119 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 1b8d0e26d25e4888bc95c0956520bdd1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260321 02:58:36.145700 11979 raft_consensus.cc:2955] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.153026 11134 catalog_manager.cc:5671] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:36.178318 11217 tablet_service.cc:1558] Processing DeleteTablet for tablet 1b8d0e26d25e4888bc95c0956520bdd1 with delete_type TABLET_DATA_TOMBSTONED (TS 0d7efc590a1041a9bd4cd4a971332dfb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58196
I20260321 02:58:36.193109 12137 tablet_replica.cc:333] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb: stopping tablet replica
I20260321 02:58:36.194159 12137 raft_consensus.cc:2243] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:36.197136 12137 raft_consensus.cc:2272] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:36.200786 12137 ts_tablet_manager.cc:1916] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 02:58:36.223600 12137 ts_tablet_manager.cc:1929] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260321 02:58:36.224591 12137 log.cc:1199] T 1b8d0e26d25e4888bc95c0956520bdd1 P 0d7efc590a1041a9bd4cd4a971332dfb: Deleting WAL directory at /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-0-root/wals/1b8d0e26d25e4888bc95c0956520bdd1
I20260321 02:58:36.263885 11119 catalog_manager.cc:5002] TS 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): tablet 1b8d0e26d25e4888bc95c0956520bdd1 (table test-workload [id=07ae567e73b94faa9ae558018daa9713]) successfully deleted
I20260321 02:58:36.324992 12125 raft_consensus.cc:1064] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da: attempting to promote NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 to VOTER
I20260321 02:58:36.329597 12125 consensus_queue.cc:237] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:36.336385 12105 raft_consensus.cc:1064] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb: attempting to promote NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 to VOTER
I20260321 02:58:36.343487 12105 consensus_queue.cc:237] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:36.376720 11979 raft_consensus.cc:1275] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:36.378934 12082 consensus_queue.cc:1048] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:36.386233 11381 raft_consensus.cc:1275] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:36.389636 11236 raft_consensus.cc:1275] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:36.389865 12126 consensus_queue.cc:1048] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:36.392789 12125 consensus_queue.cc:1048] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:36.427503 11376 raft_consensus.cc:1275] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:36.429226 11979 raft_consensus.cc:1275] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:36.432035 12139 consensus_queue.cc:1048] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:36.434716 12134 consensus_queue.cc:1048] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:36.452075 11308 raft_consensus.cc:1275] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:36.454344 12139 consensus_queue.cc:1048] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:36.546499 12126 raft_consensus.cc:2955] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.556021 11977 raft_consensus.cc:2955] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.556512 11228 raft_consensus.cc:2955] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.566361 12105 raft_consensus.cc:1064] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb: attempting to promote NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 to VOTER
I20260321 02:58:36.569234 12105 consensus_queue.cc:237] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:36.568046 12089 raft_consensus.cc:2955] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.597512 11307 raft_consensus.cc:2955] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.602952 11384 raft_consensus.cc:2955] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.597566 11376 raft_consensus.cc:2955] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.604297 11134 catalog_manager.cc:5671] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.650019 11977 raft_consensus.cc:2955] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.667372 11134 catalog_manager.cc:5671] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.675627 11304 consensus_queue.cc:237] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:36.679148 11979 raft_consensus.cc:1275] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:36.681160 12105 consensus_queue.cc:1048] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:36.691075 11308 raft_consensus.cc:1275] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:36.695726 11978 raft_consensus.cc:1275] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:36.697600 12132 consensus_queue.cc:1048] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:36.697991 11376 raft_consensus.cc:1275] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:36.699857 12105 consensus_queue.cc:1048] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:36.712849 11228 consensus_queue.cc:237] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:36.716946 12089 consensus_queue.cc:1048] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:36.721302 12151 raft_consensus.cc:2955] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.723628 11978 raft_consensus.cc:2955] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.732931 11977 raft_consensus.cc:1275] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:36.734856 12139 consensus_queue.cc:1048] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:36.737453 11228 raft_consensus.cc:1275] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:36.752687 11134 catalog_manager.cc:5671] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:58:36.762902 11304 raft_consensus.cc:1275] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:36.775056 11119 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet a42977f759be4144a5862197a77e395d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260321 02:58:36.776103 12105 raft_consensus.cc:2955] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.777344 12151 consensus_queue.cc:1048] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:36.779518 11980 raft_consensus.cc:2955] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.788301 11366 tablet_service.cc:1558] Processing DeleteTablet for tablet a42977f759be4144a5862197a77e395d with delete_type TABLET_DATA_TOMBSTONED (TS fef7f8cc55b043039131bd9b2229f0a5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40932
I20260321 02:58:36.790313 12139 consensus_queue.cc:1048] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:36.802618 12089 raft_consensus.cc:2955] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.805505 11228 raft_consensus.cc:2955] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.814662 12156 tablet_replica.cc:333] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5: stopping tablet replica
I20260321 02:58:36.816972 11978 raft_consensus.cc:2955] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.825701 11134 catalog_manager.cc:5671] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.829806 12156 raft_consensus.cc:2243] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:36.831498 11308 raft_consensus.cc:2955] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.843420 11366 tablet_service.cc:1558] Processing DeleteTablet for tablet e4337508daeb4d598c4f6a0b4314878d with delete_type TABLET_DATA_TOMBSTONED (TS fef7f8cc55b043039131bd9b2229f0a5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40932
I20260321 02:58:36.847774 11119 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet e4337508daeb4d598c4f6a0b4314878d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260321 02:58:36.860548 12156 raft_consensus.cc:2272] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:36.875353 11134 catalog_manager.cc:5671] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.887530 12156 ts_tablet_manager.cc:1916] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 02:58:36.883204 11376 raft_consensus.cc:2955] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
W20260321 02:58:36.945533 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 91452.3us
I20260321 02:58:36.945393 11302 raft_consensus.cc:2955] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:36.978924 12156 ts_tablet_manager.cc:1929] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260321 02:58:36.979598 12156 log.cc:1199] T a42977f759be4144a5862197a77e395d P fef7f8cc55b043039131bd9b2229f0a5: Deleting WAL directory at /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-2-root/wals/a42977f759be4144a5862197a77e395d
I20260321 02:58:36.983886 12156 tablet_replica.cc:333] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5: stopping tablet replica
I20260321 02:58:36.981772 11119 catalog_manager.cc:5002] TS fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): tablet a42977f759be4144a5862197a77e395d (table test-workload [id=07ae567e73b94faa9ae558018daa9713]) successfully deleted
I20260321 02:58:36.995321 11235 consensus_queue.cc:237] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:37.000437 11978 raft_consensus.cc:1275] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:37.002533 12158 consensus_queue.cc:1048] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:36.985213 12156 raft_consensus.cc:2243] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:37.006886 12156 raft_consensus.cc:2272] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:37.008366 11376 raft_consensus.cc:1275] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:37.010449 12159 consensus_queue.cc:1048] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:37.014003 12158 raft_consensus.cc:2955] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:37.017671 11978 raft_consensus.cc:2955] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:37.018460 11376 raft_consensus.cc:2955] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:37.010751 12156 ts_tablet_manager.cc:1916] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 02:58:37.038894 11119 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 006e9dd81c0c4368bed549c97128f4ea with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260321 02:58:37.049820 11134 catalog_manager.cc:5671] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:37.064275 11299 tablet_service.cc:1558] Processing DeleteTablet for tablet 006e9dd81c0c4368bed549c97128f4ea with delete_type TABLET_DATA_TOMBSTONED (TS 83b12b883a7f4129a87dadf5a78f08da not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56610
I20260321 02:58:37.082638 12162 tablet_replica.cc:333] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da: stopping tablet replica
I20260321 02:58:37.084067 12162 raft_consensus.cc:2243] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:37.088157 12162 raft_consensus.cc:2272] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:37.096241 12162 ts_tablet_manager.cc:1916] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 02:58:37.096671 12156 ts_tablet_manager.cc:1929] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260321 02:58:37.097182 12156 log.cc:1199] T e4337508daeb4d598c4f6a0b4314878d P fef7f8cc55b043039131bd9b2229f0a5: Deleting WAL directory at /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-2-root/wals/e4337508daeb4d598c4f6a0b4314878d
I20260321 02:58:37.098898 11119 catalog_manager.cc:5002] TS fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): tablet e4337508daeb4d598c4f6a0b4314878d (table test-workload [id=07ae567e73b94faa9ae558018daa9713]) successfully deleted
I20260321 02:58:37.112320 12162 ts_tablet_manager.cc:1929] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260321 02:58:37.112968 12162 log.cc:1199] T 006e9dd81c0c4368bed549c97128f4ea P 83b12b883a7f4129a87dadf5a78f08da: Deleting WAL directory at /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-1-root/wals/006e9dd81c0c4368bed549c97128f4ea
I20260321 02:58:37.115013 11119 catalog_manager.cc:5002] TS 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): tablet 006e9dd81c0c4368bed549c97128f4ea (table test-workload [id=07ae567e73b94faa9ae558018daa9713]) successfully deleted
I20260321 02:58:38.139636 11304 consensus_queue.cc:237] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
W20260321 02:58:38.147711 11276 consensus_peers.cc:597] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b701d6135bc458c9aef1d5303344525. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:38.147804 11376 raft_consensus.cc:1275] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:38.148898 11230 raft_consensus.cc:1275] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:38.149791 12151 consensus_queue.cc:1048] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:38.150851 12132 consensus_queue.cc:1048] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:38.180157 12151 raft_consensus.cc:2955] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:38.186930 11376 raft_consensus.cc:2955] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:38.187467 11230 raft_consensus.cc:2955] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:38.206022 11304 consensus_queue.cc:237] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:38.219915 11134 catalog_manager.cc:5671] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
W20260321 02:58:38.245325 11276 consensus_peers.cc:597] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70e1ea01442f4d13a0f6432ea463c5df. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:38.246662 11381 raft_consensus.cc:1275] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:38.246662 11236 raft_consensus.cc:1275] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:38.248854 12167 consensus_queue.cc:1048] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:38.265754 12167 consensus_queue.cc:1048] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:38.276377 12151 raft_consensus.cc:2955] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:38.297055 11376 raft_consensus.cc:2955] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:38.317549 11230 raft_consensus.cc:2955] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:38.323462 11376 consensus_queue.cc:237] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
W20260321 02:58:38.347776 11350 consensus_peers.cc:597] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 709aaa09afd74a25bbd5dc62dca37f02. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:38.348325 11304 raft_consensus.cc:1275] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:38.348867 11230 raft_consensus.cc:1275] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:38.350309 12140 consensus_queue.cc:1048] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:38.351194 12142 consensus_queue.cc:1048] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:38.357448 11132 catalog_manager.cc:5671] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:38.409281 12163 raft_consensus.cc:2955] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:38.413864 11304 raft_consensus.cc:2955] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
W20260321 02:58:38.429349 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 55862.6us
I20260321 02:58:38.431267 11230 consensus_queue.cc:237] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:38.443001 11133 catalog_manager.cc:5671] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260321 02:58:38.451447 11201 consensus_peers.cc:597] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00eeadd5c566459b86e97f920a91c398. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:38.466178 11376 raft_consensus.cc:1275] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:38.468477 11302 raft_consensus.cc:1275] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:38.472990 12185 consensus_queue.cc:1048] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:38.476925 11230 raft_consensus.cc:2955] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:38.494338 12158 consensus_queue.cc:1048] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:38.508108 12185 raft_consensus.cc:2955] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:38.525056 11376 raft_consensus.cc:2955] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:38.530320 11307 raft_consensus.cc:2955] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:38.543309 11132 catalog_manager.cc:5671] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:38.580518 12188 ts_tablet_manager.cc:933] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:38.586417 12188 tablet_copy_client.cc:323] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.2:34267
I20260321 02:58:38.589998 11321 tablet_copy_service.cc:140] P 83b12b883a7f4129a87dadf5a78f08da: Received BeginTabletCopySession request for tablet 2b701d6135bc458c9aef1d5303344525 from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:49786)
I20260321 02:58:38.590647 11321 tablet_copy_service.cc:161] P 83b12b883a7f4129a87dadf5a78f08da: Beginning new tablet copy session on tablet 2b701d6135bc458c9aef1d5303344525 from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:49786: session id = 3d8d3176d6ec407ea2a965556c2e2c56-2b701d6135bc458c9aef1d5303344525
I20260321 02:58:38.599154 11321 tablet_copy_source_session.cc:215] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:58:38.603909 12188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b701d6135bc458c9aef1d5303344525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:38.621148 12188 tablet_copy_client.cc:806] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
I20260321 02:58:38.621943 12188 tablet_copy_client.cc:670] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:58:38.627076 12188 tablet_copy_client.cc:538] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 02:58:38.635140 12188 tablet_bootstrap.cc:492] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap starting.
I20260321 02:58:38.660459 12188 tablet_bootstrap.cc:492] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 02:58:38.661589 12188 tablet_bootstrap.cc:492] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap complete.
I20260321 02:58:38.662309 12188 ts_tablet_manager.cc:1403] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.012s
I20260321 02:58:38.665598 12188 raft_consensus.cc:359] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:38.666471 12188 raft_consensus.cc:740] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Initialized, Role: LEARNER
I20260321 02:58:38.667245 12188 consensus_queue.cc:260] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:38.670392 12188 ts_tablet_manager.cc:1434] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260321 02:58:38.672526 11321 tablet_copy_service.cc:342] P 83b12b883a7f4129a87dadf5a78f08da: Request end of tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-2b701d6135bc458c9aef1d5303344525 received from {username='slave'} at 127.0.0.1:49786
I20260321 02:58:38.672997 11321 tablet_copy_service.cc:434] P 83b12b883a7f4129a87dadf5a78f08da: ending tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-2b701d6135bc458c9aef1d5303344525 on tablet 2b701d6135bc458c9aef1d5303344525 with peer 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:38.679476 12188 ts_tablet_manager.cc:933] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:38.689625 12188 tablet_copy_client.cc:323] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.2:34267
I20260321 02:58:38.691236 11321 tablet_copy_service.cc:140] P 83b12b883a7f4129a87dadf5a78f08da: Received BeginTabletCopySession request for tablet 70e1ea01442f4d13a0f6432ea463c5df from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:49786)
I20260321 02:58:38.691773 11321 tablet_copy_service.cc:161] P 83b12b883a7f4129a87dadf5a78f08da: Beginning new tablet copy session on tablet 70e1ea01442f4d13a0f6432ea463c5df from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:49786: session id = 3d8d3176d6ec407ea2a965556c2e2c56-70e1ea01442f4d13a0f6432ea463c5df
I20260321 02:58:38.699606 11321 tablet_copy_source_session.cc:215] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:58:38.706784 12188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e1ea01442f4d13a0f6432ea463c5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:38.732432 12188 tablet_copy_client.cc:806] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
I20260321 02:58:38.733179 12188 tablet_copy_client.cc:670] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:58:38.740206 12188 tablet_copy_client.cc:538] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 02:58:38.751237 12188 tablet_bootstrap.cc:492] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap starting.
I20260321 02:58:38.778123 12188 tablet_bootstrap.cc:492] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 02:58:38.778923 12188 tablet_bootstrap.cc:492] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap complete.
I20260321 02:58:38.779477 12188 ts_tablet_manager.cc:1403] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent bootstrapping tablet: real 0.028s user 0.028s sys 0.001s
I20260321 02:58:38.782886 12188 raft_consensus.cc:359] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:38.783876 12188 raft_consensus.cc:740] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Initialized, Role: LEARNER
I20260321 02:58:38.784581 12188 consensus_queue.cc:260] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:38.794636 12188 ts_tablet_manager.cc:1434] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260321 02:58:38.796578 11321 tablet_copy_service.cc:342] P 83b12b883a7f4129a87dadf5a78f08da: Request end of tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-70e1ea01442f4d13a0f6432ea463c5df received from {username='slave'} at 127.0.0.1:49786
I20260321 02:58:38.797127 11321 tablet_copy_service.cc:434] P 83b12b883a7f4129a87dadf5a78f08da: ending tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-70e1ea01442f4d13a0f6432ea463c5df on tablet 70e1ea01442f4d13a0f6432ea463c5df with peer 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:38.900046 12188 ts_tablet_manager.cc:933] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717)
I20260321 02:58:38.902366 12188 tablet_copy_client.cc:323] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.1:36717
I20260321 02:58:38.906383 11246 tablet_copy_service.cc:140] P 0d7efc590a1041a9bd4cd4a971332dfb: Received BeginTabletCopySession request for tablet 00eeadd5c566459b86e97f920a91c398 from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:59202)
I20260321 02:58:38.906957 11246 tablet_copy_service.cc:161] P 0d7efc590a1041a9bd4cd4a971332dfb: Beginning new tablet copy session on tablet 00eeadd5c566459b86e97f920a91c398 from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:59202: session id = 3d8d3176d6ec407ea2a965556c2e2c56-00eeadd5c566459b86e97f920a91c398
I20260321 02:58:38.923256 11246 tablet_copy_source_session.cc:215] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:58:38.928745 12188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00eeadd5c566459b86e97f920a91c398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:38.939055 11980 raft_consensus.cc:1217] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260321 02:58:38.957212 12188 tablet_copy_client.cc:806] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
I20260321 02:58:38.957808 12188 tablet_copy_client.cc:670] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:58:38.958727 12192 ts_tablet_manager.cc:933] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:38.973520 12188 tablet_copy_client.cc:538] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 02:58:38.975131 12192 tablet_copy_client.cc:323] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.3:43533
I20260321 02:58:38.977576 11395 tablet_copy_service.cc:140] P fef7f8cc55b043039131bd9b2229f0a5: Received BeginTabletCopySession request for tablet 709aaa09afd74a25bbd5dc62dca37f02 from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:45634)
I20260321 02:58:38.978279 11395 tablet_copy_service.cc:161] P fef7f8cc55b043039131bd9b2229f0a5: Beginning new tablet copy session on tablet 709aaa09afd74a25bbd5dc62dca37f02 from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:45634: session id = 3d8d3176d6ec407ea2a965556c2e2c56-709aaa09afd74a25bbd5dc62dca37f02
I20260321 02:58:38.985253 12188 tablet_bootstrap.cc:492] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap starting.
I20260321 02:58:38.989583 11395 tablet_copy_source_session.cc:215] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:58:38.994272 12192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 709aaa09afd74a25bbd5dc62dca37f02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:39.035161 12188 tablet_bootstrap.cc:492] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 02:58:39.036207 12188 tablet_bootstrap.cc:492] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap complete.
I20260321 02:58:39.036401 12192 tablet_copy_client.cc:806] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
I20260321 02:58:39.036962 12188 ts_tablet_manager.cc:1403] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent bootstrapping tablet: real 0.052s user 0.015s sys 0.008s
I20260321 02:58:39.037145 12192 tablet_copy_client.cc:670] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:58:39.039638 12188 raft_consensus.cc:359] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:39.040577 12188 raft_consensus.cc:740] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Initialized, Role: LEARNER
I20260321 02:58:39.041205 12188 consensus_queue.cc:260] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:39.043229 12192 tablet_copy_client.cc:538] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 02:58:39.048650 12188 ts_tablet_manager.cc:1434] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260321 02:58:39.051206 11246 tablet_copy_service.cc:342] P 0d7efc590a1041a9bd4cd4a971332dfb: Request end of tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-00eeadd5c566459b86e97f920a91c398 received from {username='slave'} at 127.0.0.1:59202
I20260321 02:58:39.051790 11246 tablet_copy_service.cc:434] P 0d7efc590a1041a9bd4cd4a971332dfb: ending tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-00eeadd5c566459b86e97f920a91c398 on tablet 00eeadd5c566459b86e97f920a91c398 with peer 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:39.061120 12192 tablet_bootstrap.cc:492] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap starting.
I20260321 02:58:39.096459 12192 tablet_bootstrap.cc:492] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 02:58:39.097445 12192 tablet_bootstrap.cc:492] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap complete.
I20260321 02:58:39.098155 12192 ts_tablet_manager.cc:1403] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent bootstrapping tablet: real 0.037s user 0.023s sys 0.007s
I20260321 02:58:39.101468 11980 raft_consensus.cc:1217] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260321 02:58:39.124614 12192 raft_consensus.cc:359] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:39.131420 12192 raft_consensus.cc:740] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Initialized, Role: LEARNER
I20260321 02:58:39.133240 12192 consensus_queue.cc:260] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:39.139273 12192 ts_tablet_manager.cc:1434] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent starting tablet: real 0.041s user 0.005s sys 0.004s
I20260321 02:58:39.140995 11395 tablet_copy_service.cc:342] P fef7f8cc55b043039131bd9b2229f0a5: Request end of tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-709aaa09afd74a25bbd5dc62dca37f02 received from {username='slave'} at 127.0.0.1:45634
I20260321 02:58:39.141496 11395 tablet_copy_service.cc:434] P fef7f8cc55b043039131bd9b2229f0a5: ending tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-709aaa09afd74a25bbd5dc62dca37f02 on tablet 709aaa09afd74a25bbd5dc62dca37f02 with peer 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:39.279140 11980 raft_consensus.cc:1217] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260321 02:58:39.453874 11980 raft_consensus.cc:1217] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260321 02:58:39.512238 12173 raft_consensus.cc:1064] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da: attempting to promote NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 to VOTER
I20260321 02:58:39.514815 12173 consensus_queue.cc:237] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:39.521349 11980 raft_consensus.cc:1275] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:39.521512 11381 raft_consensus.cc:1275] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:39.522974 12167 consensus_queue.cc:1048] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:39.525457 12167 consensus_queue.cc:1048] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:39.525388 11236 raft_consensus.cc:1275] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:39.528669 12174 consensus_queue.cc:1048] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:39.546231 12167 raft_consensus.cc:2955] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.549335 11381 raft_consensus.cc:2955] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.584285 11980 raft_consensus.cc:2955] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.615805 11132 catalog_manager.cc:5671] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:58:39.630820 11230 raft_consensus.cc:2955] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.653371 12167 raft_consensus.cc:1064] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da: attempting to promote NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 to VOTER
I20260321 02:58:39.657444 12167 consensus_queue.cc:237] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:39.678308 11980 raft_consensus.cc:1275] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:39.679597 12167 consensus_queue.cc:1048] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:39.680927 11236 raft_consensus.cc:1275] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:39.682554 11376 raft_consensus.cc:1275] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:39.685250 12203 consensus_queue.cc:1048] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:39.688333 12174 consensus_queue.cc:1048] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:39.706158 12180 raft_consensus.cc:1064] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb: attempting to promote NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 to VOTER
I20260321 02:58:39.713614 12180 consensus_queue.cc:237] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:39.723783 12203 raft_consensus.cc:2955] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.729050 11980 raft_consensus.cc:2955] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.730036 11376 raft_consensus.cc:1275] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:39.738010 11979 raft_consensus.cc:1275] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:39.738832 12158 consensus_queue.cc:1048] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:39.740134 12185 consensus_queue.cc:1048] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:39.746296 11304 consensus_queue.cc:237] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:39.746177 11230 raft_consensus.cc:2955] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.755239 11978 raft_consensus.cc:1275] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:39.756999 12167 consensus_queue.cc:1048] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:39.758694 11376 raft_consensus.cc:1275] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:39.763688 12167 consensus_queue.cc:1048] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:39.765331 12173 raft_consensus.cc:2955] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.767531 11978 raft_consensus.cc:2955] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.768805 11133 catalog_manager.cc:5671] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.773065 11381 raft_consensus.cc:2955] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.773140 11376 raft_consensus.cc:2955] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.792842 11119 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 2b701d6135bc458c9aef1d5303344525 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260321 02:58:39.804687 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 72039.1us
I20260321 02:58:39.806993 11302 raft_consensus.cc:1275] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:39.810248 11133 catalog_manager.cc:5671] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.814188 11304 consensus_queue.cc:237] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:39.821017 12215 consensus_queue.cc:1048] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:39.833634 11384 raft_consensus.cc:1275] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:39.835523 11978 raft_consensus.cc:1275] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:39.837293 12173 consensus_queue.cc:1048] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:39.838279 12180 raft_consensus.cc:2955] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.843739 11384 raft_consensus.cc:2955] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.844285 12203 raft_consensus.cc:2955] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.846021 11979 raft_consensus.cc:2955] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.849169 11217 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b701d6135bc458c9aef1d5303344525 with delete_type TABLET_DATA_TOMBSTONED (TS 0d7efc590a1041a9bd4cd4a971332dfb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58196
I20260321 02:58:39.849572 12167 consensus_queue.cc:1048] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:39.856598 11381 raft_consensus.cc:2955] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.871734 12219 tablet_replica.cc:333] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb: stopping tablet replica
I20260321 02:58:39.877880 11119 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 70e1ea01442f4d13a0f6432ea463c5df with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260321 02:58:39.886196 11978 raft_consensus.cc:2955] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.895761 11134 catalog_manager.cc:5671] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.900429 12219 raft_consensus.cc:2243] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:39.878754 11133 catalog_manager.cc:5671] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.904646 12219 raft_consensus.cc:2272] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:39.908443 11307 raft_consensus.cc:2955] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:39.913853 12219 ts_tablet_manager.cc:1916] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 02:58:39.940205 11217 tablet_service.cc:1558] Processing DeleteTablet for tablet 70e1ea01442f4d13a0f6432ea463c5df with delete_type TABLET_DATA_TOMBSTONED (TS 0d7efc590a1041a9bd4cd4a971332dfb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58196
I20260321 02:58:39.959446 12219 ts_tablet_manager.cc:1929] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260321 02:58:39.961045 12219 log.cc:1199] T 2b701d6135bc458c9aef1d5303344525 P 0d7efc590a1041a9bd4cd4a971332dfb: Deleting WAL directory at /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-0-root/wals/2b701d6135bc458c9aef1d5303344525
I20260321 02:58:39.964035 12219 tablet_replica.cc:333] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb: stopping tablet replica
I20260321 02:58:39.964393 11119 catalog_manager.cc:5002] TS 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): tablet 2b701d6135bc458c9aef1d5303344525 (table test-workload [id=07ae567e73b94faa9ae558018daa9713]) successfully deleted
I20260321 02:58:39.967250 12140 raft_consensus.cc:1064] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5: attempting to promote NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 to VOTER
I20260321 02:58:39.969414 11236 consensus_queue.cc:237] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:39.971527 12140 consensus_queue.cc:237] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:39.982758 11978 raft_consensus.cc:1275] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:39.986949 11307 raft_consensus.cc:1275] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:39.986115 12219 raft_consensus.cc:2243] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:39.988659 12219 raft_consensus.cc:2272] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:39.989941 12180 consensus_queue.cc:1048] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:39.992594 12219 ts_tablet_manager.cc:1916] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 02:58:40.007336 11979 raft_consensus.cc:1275] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:40.008832 11307 raft_consensus.cc:1275] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:40.010624 12140 consensus_queue.cc:1048] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:40.013661 12163 consensus_queue.cc:1048] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:40.018225 12219 ts_tablet_manager.cc:1929] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260321 02:58:40.018970 12219 log.cc:1199] T 70e1ea01442f4d13a0f6432ea463c5df P 0d7efc590a1041a9bd4cd4a971332dfb: Deleting WAL directory at /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-0-root/wals/70e1ea01442f4d13a0f6432ea463c5df
I20260321 02:58:40.029870 11119 catalog_manager.cc:5002] TS 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): tablet 70e1ea01442f4d13a0f6432ea463c5df (table test-workload [id=07ae567e73b94faa9ae558018daa9713]) successfully deleted
W20260321 02:58:40.036644 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 48493.6us
I20260321 02:58:40.043989 12185 consensus_queue.cc:1048] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:40.047652 11978 raft_consensus.cc:2955] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:40.046046 12220 raft_consensus.cc:2955] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:40.054800 11236 raft_consensus.cc:1275] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:40.077980 11119 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 00eeadd5c566459b86e97f920a91c398 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260321 02:58:40.088945 12163 consensus_queue.cc:1048] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:40.090574 11302 raft_consensus.cc:2955] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:40.094568 11133 catalog_manager.cc:5671] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:58:40.121558 12204 raft_consensus.cc:2955] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:40.124683 11366 tablet_service.cc:1558] Processing DeleteTablet for tablet 00eeadd5c566459b86e97f920a91c398 with delete_type TABLET_DATA_TOMBSTONED (TS fef7f8cc55b043039131bd9b2229f0a5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40932
I20260321 02:58:40.135326 11307 raft_consensus.cc:2955] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:40.135915 11230 raft_consensus.cc:2955] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:40.151147 11978 raft_consensus.cc:2955] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:40.163444 12226 tablet_replica.cc:333] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5: stopping tablet replica
I20260321 02:58:40.167766 12226 raft_consensus.cc:2243] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:40.170570 12226 raft_consensus.cc:2272] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:40.159830 11134 catalog_manager.cc:5671] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:40.179217 12226 ts_tablet_manager.cc:1916] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 02:58:40.199590 11376 consensus_queue.cc:237] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:40.206934 11978 raft_consensus.cc:1275] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:40.208405 12140 consensus_queue.cc:1048] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:40.231482 12206 raft_consensus.cc:2955] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:40.232653 11980 raft_consensus.cc:2955] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:40.247519 12226 ts_tablet_manager.cc:1929] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260321 02:58:40.248159 12226 log.cc:1199] T 00eeadd5c566459b86e97f920a91c398 P fef7f8cc55b043039131bd9b2229f0a5: Deleting WAL directory at /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-2-root/wals/00eeadd5c566459b86e97f920a91c398
I20260321 02:58:40.265455 11230 raft_consensus.cc:1275] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:40.260144 11119 catalog_manager.cc:5002] TS fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): tablet 00eeadd5c566459b86e97f920a91c398 (table test-workload [id=07ae567e73b94faa9ae558018daa9713]) successfully deleted
W20260321 02:58:40.440248 11119 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 180115us
I20260321 02:58:40.454876 11119 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 709aaa09afd74a25bbd5dc62dca37f02 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260321 02:58:40.469741 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 201388us
I20260321 02:58:40.474395 12228 consensus_queue.cc:1048] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:40.498116 11233 raft_consensus.cc:2955] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:40.695688 11133 catalog_manager.cc:5671] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
W20260321 02:58:40.724964 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 199106us
I20260321 02:58:40.805657 11299 tablet_service.cc:1558] Processing DeleteTablet for tablet 709aaa09afd74a25bbd5dc62dca37f02 with delete_type TABLET_DATA_TOMBSTONED (TS 83b12b883a7f4129a87dadf5a78f08da not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56610
I20260321 02:58:40.807752 12236 tablet_replica.cc:333] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da: stopping tablet replica
I20260321 02:58:40.828770 12236 raft_consensus.cc:2243] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:40.831152 12236 raft_consensus.cc:2272] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:40.837733 12236 ts_tablet_manager.cc:1916] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 02:58:40.861131 12236 ts_tablet_manager.cc:1929] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260321 02:58:40.861637 12236 log.cc:1199] T 709aaa09afd74a25bbd5dc62dca37f02 P 83b12b883a7f4129a87dadf5a78f08da: Deleting WAL directory at /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-1-root/wals/709aaa09afd74a25bbd5dc62dca37f02
I20260321 02:58:40.863643 11119 catalog_manager.cc:5002] TS 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): tablet 709aaa09afd74a25bbd5dc62dca37f02 (table test-workload [id=07ae567e73b94faa9ae558018daa9713]) successfully deleted
I20260321 02:58:41.531559 11307 consensus_queue.cc:237] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:41.539525 11231 raft_consensus.cc:1275] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260321 02:58:41.540117 11276 consensus_peers.cc:597] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aab7e035f7984ed8ba6560e1591ec831. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:41.541802 11377 raft_consensus.cc:1275] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:41.542675 12238 consensus_queue.cc:1048] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:41.544543 12173 consensus_queue.cc:1048] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:41.554139 12203 raft_consensus.cc:2955] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:41.555572 11231 raft_consensus.cc:2955] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
W20260321 02:58:41.558979 11257 debug-util.cc:398] Leaking SignalData structure 0x7b080018f1a0 after lost signal to thread 10071
W20260321 02:58:41.560068 11257 debug-util.cc:398] Leaking SignalData structure 0x7b08001bee40 after lost signal to thread 11168
W20260321 02:58:41.561237 11257 debug-util.cc:398] Leaking SignalData structure 0x7b080031b820 after lost signal to thread 11260
W20260321 02:58:41.562258 11257 debug-util.cc:398] Leaking SignalData structure 0x7b0800263180 after lost signal to thread 11335
W20260321 02:58:41.563377 11257 debug-util.cc:398] Leaking SignalData structure 0x7b0800103360 after lost signal to thread 11409
W20260321 02:58:41.566074 11257 debug-util.cc:398] Leaking SignalData structure 0x7b0800305960 after lost signal to thread 12004
I20260321 02:58:41.577515 11307 consensus_queue.cc:237] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:41.582535 11382 raft_consensus.cc:2955] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:41.579078 11134 catalog_manager.cc:5671] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
W20260321 02:58:41.593602 11276 consensus_peers.cc:597] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd04df7a33b14699ab2d2bd6325942e4. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:41.595600 11377 raft_consensus.cc:1275] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:41.599000 12203 consensus_queue.cc:1048] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:41.599624 11231 raft_consensus.cc:1275] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:41.601632 12238 consensus_queue.cc:1048] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:41.628281 12203 raft_consensus.cc:2955] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:41.640655 11380 raft_consensus.cc:2955] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:41.642797 11231 raft_consensus.cc:2955] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:41.675204 11231 consensus_queue.cc:237] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:41.677953 11134 catalog_manager.cc:5671] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:41.702057 11380 raft_consensus.cc:1275] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:41.706277 12158 consensus_queue.cc:1048] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:41.708249 11307 raft_consensus.cc:1275] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:41.714879 12185 consensus_queue.cc:1048] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260321 02:58:41.720402 11201 consensus_peers.cc:597] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 515945e5685c4afe9d41d5317acb434c. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:41.736403 12220 raft_consensus.cc:2955] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:41.739035 11382 raft_consensus.cc:2955] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:41.777052 11231 consensus_queue.cc:237] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:41.781613 11307 raft_consensus.cc:2955] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:41.785866 11133 catalog_manager.cc:5671] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:41.827926 11382 raft_consensus.cc:1275] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:41.830884 12185 consensus_queue.cc:1048] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260321 02:58:41.832130 11201 consensus_peers.cc:597] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fcb8dea530d4005a0101183a89b709d. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:41.832898 11307 raft_consensus.cc:1275] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:41.835273 12158 consensus_queue.cc:1048] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:41.846796 12185 raft_consensus.cc:2955] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:41.856598 11377 raft_consensus.cc:2955] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:41.858644 11307 raft_consensus.cc:2955] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:41.897318 11134 catalog_manager.cc:5671] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:41.944931 12256 ts_tablet_manager.cc:933] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:41.948729 12256 tablet_copy_client.cc:323] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.2:34267
I20260321 02:58:41.950909 11321 tablet_copy_service.cc:140] P 83b12b883a7f4129a87dadf5a78f08da: Received BeginTabletCopySession request for tablet aab7e035f7984ed8ba6560e1591ec831 from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:49786)
I20260321 02:58:41.951768 11321 tablet_copy_service.cc:161] P 83b12b883a7f4129a87dadf5a78f08da: Beginning new tablet copy session on tablet aab7e035f7984ed8ba6560e1591ec831 from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:49786: session id = 3d8d3176d6ec407ea2a965556c2e2c56-aab7e035f7984ed8ba6560e1591ec831
I20260321 02:58:41.959779 11321 tablet_copy_source_session.cc:215] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:58:41.966395 12256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aab7e035f7984ed8ba6560e1591ec831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:41.988801 12256 tablet_copy_client.cc:806] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
I20260321 02:58:41.989621 12256 tablet_copy_client.cc:670] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:58:41.993942 12256 tablet_copy_client.cc:538] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 02:58:42.005941 12256 tablet_bootstrap.cc:492] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap starting.
I20260321 02:58:42.034662 12256 tablet_bootstrap.cc:492] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 02:58:42.035674 12256 tablet_bootstrap.cc:492] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap complete.
I20260321 02:58:42.036458 12256 ts_tablet_manager.cc:1403] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.015s
I20260321 02:58:42.039371 12256 raft_consensus.cc:359] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:42.040323 12256 raft_consensus.cc:740] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Initialized, Role: LEARNER
I20260321 02:58:42.041042 12256 consensus_queue.cc:260] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:42.050926 12256 ts_tablet_manager.cc:1434] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent starting tablet: real 0.014s user 0.006s sys 0.005s
I20260321 02:58:42.052935 11321 tablet_copy_service.cc:342] P 83b12b883a7f4129a87dadf5a78f08da: Request end of tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-aab7e035f7984ed8ba6560e1591ec831 received from {username='slave'} at 127.0.0.1:49786
I20260321 02:58:42.053419 11321 tablet_copy_service.cc:434] P 83b12b883a7f4129a87dadf5a78f08da: ending tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-aab7e035f7984ed8ba6560e1591ec831 on tablet aab7e035f7984ed8ba6560e1591ec831 with peer 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:42.070134 12256 ts_tablet_manager.cc:933] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:42.157977 12256 tablet_copy_client.cc:323] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.2:34267
I20260321 02:58:42.159581 11321 tablet_copy_service.cc:140] P 83b12b883a7f4129a87dadf5a78f08da: Received BeginTabletCopySession request for tablet bd04df7a33b14699ab2d2bd6325942e4 from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:49786)
I20260321 02:58:42.160187 11321 tablet_copy_service.cc:161] P 83b12b883a7f4129a87dadf5a78f08da: Beginning new tablet copy session on tablet bd04df7a33b14699ab2d2bd6325942e4 from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:49786: session id = 3d8d3176d6ec407ea2a965556c2e2c56-bd04df7a33b14699ab2d2bd6325942e4
I20260321 02:58:42.173398 11321 tablet_copy_source_session.cc:215] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:58:42.186081 12256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd04df7a33b14699ab2d2bd6325942e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:42.208274 12256 tablet_copy_client.cc:806] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
I20260321 02:58:42.209000 12256 tablet_copy_client.cc:670] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:58:42.213281 12256 tablet_copy_client.cc:538] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 02:58:42.226202 12256 tablet_bootstrap.cc:492] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap starting.
I20260321 02:58:42.242266 12261 ts_tablet_manager.cc:933] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717)
I20260321 02:58:42.259629 12256 tablet_bootstrap.cc:492] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 02:58:42.260607 12256 tablet_bootstrap.cc:492] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap complete.
I20260321 02:58:42.261281 12256 ts_tablet_manager.cc:1403] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent bootstrapping tablet: real 0.035s user 0.016s sys 0.010s
I20260321 02:58:42.264034 12256 raft_consensus.cc:359] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:42.265048 12256 raft_consensus.cc:740] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Initialized, Role: LEARNER
I20260321 02:58:42.265678 12256 consensus_queue.cc:260] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:42.268148 12256 ts_tablet_manager.cc:1434] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260321 02:58:42.272172 11321 tablet_copy_service.cc:342] P 83b12b883a7f4129a87dadf5a78f08da: Request end of tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-bd04df7a33b14699ab2d2bd6325942e4 received from {username='slave'} at 127.0.0.1:49786
I20260321 02:58:42.272665 11321 tablet_copy_service.cc:434] P 83b12b883a7f4129a87dadf5a78f08da: ending tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-bd04df7a33b14699ab2d2bd6325942e4 on tablet bd04df7a33b14699ab2d2bd6325942e4 with peer 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:42.272212 12261 tablet_copy_client.cc:323] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.1:36717
I20260321 02:58:42.280364 11246 tablet_copy_service.cc:140] P 0d7efc590a1041a9bd4cd4a971332dfb: Received BeginTabletCopySession request for tablet 515945e5685c4afe9d41d5317acb434c from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:59202)
I20260321 02:58:42.281006 11246 tablet_copy_service.cc:161] P 0d7efc590a1041a9bd4cd4a971332dfb: Beginning new tablet copy session on tablet 515945e5685c4afe9d41d5317acb434c from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:59202: session id = 3d8d3176d6ec407ea2a965556c2e2c56-515945e5685c4afe9d41d5317acb434c
I20260321 02:58:42.292213 11246 tablet_copy_source_session.cc:215] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:58:42.302362 12261 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 515945e5685c4afe9d41d5317acb434c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:42.325243 12261 tablet_copy_client.cc:806] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
I20260321 02:58:42.325918 12261 tablet_copy_client.cc:670] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:58:42.335997 12261 tablet_copy_client.cc:538] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 02:58:42.345261 12261 tablet_bootstrap.cc:492] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap starting.
I20260321 02:58:42.347774 12256 ts_tablet_manager.cc:933] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717)
I20260321 02:58:42.366861 12256 tablet_copy_client.cc:323] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.1:36717
I20260321 02:58:42.371997 11246 tablet_copy_service.cc:140] P 0d7efc590a1041a9bd4cd4a971332dfb: Received BeginTabletCopySession request for tablet 2fcb8dea530d4005a0101183a89b709d from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:59202)
I20260321 02:58:42.372613 11246 tablet_copy_service.cc:161] P 0d7efc590a1041a9bd4cd4a971332dfb: Beginning new tablet copy session on tablet 2fcb8dea530d4005a0101183a89b709d from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:59202: session id = 3d8d3176d6ec407ea2a965556c2e2c56-2fcb8dea530d4005a0101183a89b709d
I20260321 02:58:42.376956 12261 tablet_bootstrap.cc:492] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 02:58:42.378022 12261 tablet_bootstrap.cc:492] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap complete.
I20260321 02:58:42.378737 12261 ts_tablet_manager.cc:1403] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.011s
I20260321 02:58:42.382452 12261 raft_consensus.cc:359] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:42.383564 12261 raft_consensus.cc:740] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Initialized, Role: LEARNER
I20260321 02:58:42.386615 12261 consensus_queue.cc:260] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:42.389645 12261 ts_tablet_manager.cc:1434] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent starting tablet: real 0.011s user 0.004s sys 0.004s
I20260321 02:58:42.395318 11245 tablet_copy_service.cc:342] P 0d7efc590a1041a9bd4cd4a971332dfb: Request end of tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-515945e5685c4afe9d41d5317acb434c received from {username='slave'} at 127.0.0.1:59202
I20260321 02:58:42.395821 11245 tablet_copy_service.cc:434] P 0d7efc590a1041a9bd4cd4a971332dfb: ending tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-515945e5685c4afe9d41d5317acb434c on tablet 515945e5685c4afe9d41d5317acb434c with peer 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:42.400810 11246 tablet_copy_source_session.cc:215] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:58:42.411257 12256 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fcb8dea530d4005a0101183a89b709d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:42.419127 11980 raft_consensus.cc:1217] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260321 02:58:42.454279 12256 tablet_copy_client.cc:806] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
I20260321 02:58:42.456066 12256 tablet_copy_client.cc:670] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:58:42.460906 12256 tablet_copy_client.cc:538] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 02:58:42.475011 12256 tablet_bootstrap.cc:492] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap starting.
I20260321 02:58:42.494628 12256 tablet_bootstrap.cc:492] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 02:58:42.495491 12256 tablet_bootstrap.cc:492] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap complete.
I20260321 02:58:42.496122 12256 ts_tablet_manager.cc:1403] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent bootstrapping tablet: real 0.022s user 0.022s sys 0.000s
I20260321 02:58:42.498860 12256 raft_consensus.cc:359] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:42.499648 12256 raft_consensus.cc:740] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Initialized, Role: LEARNER
I20260321 02:58:42.500316 12256 consensus_queue.cc:260] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:42.502734 12256 ts_tablet_manager.cc:1434] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260321 02:58:42.504474 11246 tablet_copy_service.cc:342] P 0d7efc590a1041a9bd4cd4a971332dfb: Request end of tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-2fcb8dea530d4005a0101183a89b709d received from {username='slave'} at 127.0.0.1:59202
I20260321 02:58:42.504907 11246 tablet_copy_service.cc:434] P 0d7efc590a1041a9bd4cd4a971332dfb: ending tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-2fcb8dea530d4005a0101183a89b709d on tablet 2fcb8dea530d4005a0101183a89b709d with peer 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:42.629595 11980 raft_consensus.cc:1217] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260321 02:58:42.769028 11980 raft_consensus.cc:1217] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260321 02:58:42.851696 12203 raft_consensus.cc:1064] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da: attempting to promote NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 to VOTER
I20260321 02:58:42.855652 12203 consensus_queue.cc:237] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:42.868475 11980 raft_consensus.cc:1275] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:42.869345 11231 raft_consensus.cc:1275] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:42.869993 11382 raft_consensus.cc:1275] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:42.872059 12247 consensus_queue.cc:1048] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:42.873023 12268 consensus_queue.cc:1048] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:42.874251 12248 consensus_queue.cc:1048] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:42.909947 12247 raft_consensus.cc:2955] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:42.912937 11382 raft_consensus.cc:2955] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:42.914741 11229 raft_consensus.cc:2955] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:42.938347 11980 raft_consensus.cc:2955] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:42.948508 11134 catalog_manager.cc:5671] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:58:42.953347 11979 raft_consensus.cc:1217] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260321 02:58:43.055640 12268 raft_consensus.cc:1064] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da: attempting to promote NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 to VOTER
I20260321 02:58:43.058560 12268 consensus_queue.cc:237] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:43.076876 11980 raft_consensus.cc:1275] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:43.081321 12248 consensus_queue.cc:1048] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:43.082262 11382 raft_consensus.cc:1275] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:43.082362 11231 raft_consensus.cc:1275] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:43.088654 12268 consensus_queue.cc:1048] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:43.090022 12203 consensus_queue.cc:1048] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:43.114557 11307 consensus_queue.cc:237] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:43.140808 12247 raft_consensus.cc:2955] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.145874 11229 raft_consensus.cc:2955] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.147944 11980 raft_consensus.cc:1275] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:43.149379 11382 raft_consensus.cc:1275] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:43.153780 12268 consensus_queue.cc:1048] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:43.150713 11377 raft_consensus.cc:2955] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.169437 12248 raft_consensus.cc:2955] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.173789 11132 catalog_manager.cc:5671] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.178086 12203 consensus_queue.cc:1048] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:43.219194 11119 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet aab7e035f7984ed8ba6560e1591ec831 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260321 02:58:43.223244 11980 raft_consensus.cc:2955] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.237609 11382 raft_consensus.cc:2955] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.239065 11979 raft_consensus.cc:2955] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.256930 11132 catalog_manager.cc:5671] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.295639 11217 tablet_service.cc:1558] Processing DeleteTablet for tablet aab7e035f7984ed8ba6560e1591ec831 with delete_type TABLET_DATA_TOMBSTONED (TS 0d7efc590a1041a9bd4cd4a971332dfb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58196
I20260321 02:58:43.310959 11303 consensus_queue.cc:237] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:43.332360 11979 raft_consensus.cc:1275] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:43.333933 12281 consensus_queue.cc:1048] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:43.336424 11231 raft_consensus.cc:1275] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:43.338186 12284 tablet_replica.cc:333] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb: stopping tablet replica
I20260321 02:58:43.338709 12281 consensus_queue.cc:1048] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:43.346625 12283 raft_consensus.cc:2955] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.352957 12284 raft_consensus.cc:2243] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:43.353736 12284 raft_consensus.cc:2272] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:43.352859 11979 raft_consensus.cc:2955] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.363555 12284 ts_tablet_manager.cc:1916] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 02:58:43.384161 11131 catalog_manager.cc:5671] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.385516 12158 raft_consensus.cc:1064] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb: attempting to promote NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 to VOTER
I20260321 02:58:43.387567 12284 ts_tablet_manager.cc:1929] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260321 02:58:43.388047 12284 log.cc:1199] T aab7e035f7984ed8ba6560e1591ec831 P 0d7efc590a1041a9bd4cd4a971332dfb: Deleting WAL directory at /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-0-root/wals/aab7e035f7984ed8ba6560e1591ec831
I20260321 02:58:43.388646 12158 consensus_queue.cc:237] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:43.398005 11119 catalog_manager.cc:5002] TS 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): tablet aab7e035f7984ed8ba6560e1591ec831 (table test-workload [id=07ae567e73b94faa9ae558018daa9713]) successfully deleted
I20260321 02:58:43.371460 11229 raft_consensus.cc:2955] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.402911 11366 tablet_service.cc:1558] Processing DeleteTablet for tablet bd04df7a33b14699ab2d2bd6325942e4 with delete_type TABLET_DATA_TOMBSTONED (TS fef7f8cc55b043039131bd9b2229f0a5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40932
I20260321 02:58:43.415452 12286 tablet_replica.cc:333] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5: stopping tablet replica
I20260321 02:58:43.416886 12286 raft_consensus.cc:2243] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:43.417230 12185 raft_consensus.cc:1064] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb: attempting to promote NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 to VOTER
I20260321 02:58:43.418406 12286 raft_consensus.cc:2272] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:43.420106 12185 consensus_queue.cc:237] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:43.423516 12286 ts_tablet_manager.cc:1916] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 02:58:43.428474 11980 raft_consensus.cc:1275] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:43.430562 12220 consensus_queue.cc:1048] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:43.434072 11377 raft_consensus.cc:1275] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:43.434116 11382 raft_consensus.cc:1275] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:43.436586 11980 raft_consensus.cc:1275] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:43.439086 11304 raft_consensus.cc:1275] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:43.440102 12185 consensus_queue.cc:1048] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:43.441305 12158 consensus_queue.cc:1048] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:43.442816 12220 consensus_queue.cc:1048] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:43.463898 11119 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet bd04df7a33b14699ab2d2bd6325942e4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260321 02:58:43.473631 12290 consensus_queue.cc:1048] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:43.477077 11303 raft_consensus.cc:1275] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:43.488119 12158 consensus_queue.cc:1048] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:43.501016 12185 raft_consensus.cc:2955] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.517540 11379 raft_consensus.cc:2955] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.547048 11131 catalog_manager.cc:5671] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.549078 11979 raft_consensus.cc:2955] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.560525 12286 ts_tablet_manager.cc:1929] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260321 02:58:43.565165 12288 raft_consensus.cc:2955] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.561012 12286 log.cc:1199] T bd04df7a33b14699ab2d2bd6325942e4 P fef7f8cc55b043039131bd9b2229f0a5: Deleting WAL directory at /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-2-root/wals/bd04df7a33b14699ab2d2bd6325942e4
W20260321 02:58:43.640780 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 87178.6us
I20260321 02:58:43.642364 11229 consensus_queue.cc:237] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:43.649050 11119 catalog_manager.cc:5002] TS fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): tablet bd04df7a33b14699ab2d2bd6325942e4 (table test-workload [id=07ae567e73b94faa9ae558018daa9713]) successfully deleted
I20260321 02:58:43.656878 11979 raft_consensus.cc:1275] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:43.660414 11979 raft_consensus.cc:2955] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.661839 11377 raft_consensus.cc:2955] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.669334 11308 raft_consensus.cc:2955] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.671411 12295 consensus_queue.cc:1048] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20260321 02:58:43.690789 11131 catalog_manager.cc:5671] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.700318 11379 raft_consensus.cc:1275] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:43.722081 11230 consensus_queue.cc:237] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:43.724416 12158 raft_consensus.cc:2955] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.729038 11979 raft_consensus.cc:1275] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:43.732935 11978 raft_consensus.cc:2955] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.740270 12288 consensus_queue.cc:1048] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:43.751040 12297 raft_consensus.cc:2955] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.758541 11979 raft_consensus.cc:2955] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.759883 11119 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 515945e5685c4afe9d41d5317acb434c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260321 02:58:43.765571 11119 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 2fcb8dea530d4005a0101183a89b709d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260321 02:58:43.784117 11132 catalog_manager.cc:5671] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.794680 12297 consensus_queue.cc:1048] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:43.802273 11299 tablet_service.cc:1558] Processing DeleteTablet for tablet 515945e5685c4afe9d41d5317acb434c with delete_type TABLET_DATA_TOMBSTONED (TS 83b12b883a7f4129a87dadf5a78f08da not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56610
I20260321 02:58:43.805873 11377 raft_consensus.cc:1275] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:43.808179 11131 catalog_manager.cc:5671] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.825367 11301 tablet_service.cc:1558] Processing DeleteTablet for tablet 2fcb8dea530d4005a0101183a89b709d with delete_type TABLET_DATA_TOMBSTONED (TS 83b12b883a7f4129a87dadf5a78f08da not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56610
I20260321 02:58:43.835655 11377 raft_consensus.cc:2955] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.862437 12300 tablet_replica.cc:333] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da: stopping tablet replica
I20260321 02:58:43.864832 12300 raft_consensus.cc:2243] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:43.866189 12300 pending_rounds.cc:70] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da: Trying to abort 1 pending ops.
I20260321 02:58:43.867106 12300 pending_rounds.cc:77] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7266557638191710208 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "515945e5685c4afe9d41d5317acb434c" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } } }
I20260321 02:58:43.869398 12300 raft_consensus.cc:2917] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20260321 02:58:43.870126 12300 raft_consensus.cc:2889] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260321 02:58:43.870995 12288 consensus_queue.cc:1048] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:43.870627 12300 raft_consensus.cc:2272] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:43.881549 12300 ts_tablet_manager.cc:1916] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 02:58:43.880913 11379 raft_consensus.cc:2955] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:43.902499 12300 ts_tablet_manager.cc:1929] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260321 02:58:43.903112 12300 log.cc:1199] T 515945e5685c4afe9d41d5317acb434c P 83b12b883a7f4129a87dadf5a78f08da: Deleting WAL directory at /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-1-root/wals/515945e5685c4afe9d41d5317acb434c
I20260321 02:58:43.904831 11119 catalog_manager.cc:5002] TS 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): tablet 515945e5685c4afe9d41d5317acb434c (table test-workload [id=07ae567e73b94faa9ae558018daa9713]) successfully deleted
I20260321 02:58:43.904999 12300 tablet_replica.cc:333] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da: stopping tablet replica
I20260321 02:58:43.906136 12300 raft_consensus.cc:2243] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:43.906912 12300 raft_consensus.cc:2272] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:43.910472 12300 ts_tablet_manager.cc:1916] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 02:58:43.942624 12300 ts_tablet_manager.cc:1929] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260321 02:58:43.943055 12300 log.cc:1199] T 2fcb8dea530d4005a0101183a89b709d P 83b12b883a7f4129a87dadf5a78f08da: Deleting WAL directory at /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-1-root/wals/2fcb8dea530d4005a0101183a89b709d
I20260321 02:58:43.944581 11119 catalog_manager.cc:5002] TS 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): tablet 2fcb8dea530d4005a0101183a89b709d (table test-workload [id=07ae567e73b94faa9ae558018daa9713]) successfully deleted
I20260321 02:58:44.884047 11303 consensus_queue.cc:237] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
W20260321 02:58:44.900858 11276 consensus_peers.cc:597] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76097e48b77445aaa1ffe490b8bb41cf. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:44.900877 11379 raft_consensus.cc:1275] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:44.901870 11233 raft_consensus.cc:1275] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:44.902510 12302 consensus_queue.cc:1048] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:44.904104 12281 consensus_queue.cc:1048] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:44.918591 12281 raft_consensus.cc:2955] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:44.920306 11379 raft_consensus.cc:2955] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:44.923817 11233 raft_consensus.cc:2955] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:44.937502 11377 consensus_queue.cc:237] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:44.944125 11134 catalog_manager.cc:5671] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260321 02:58:44.953852 11350 consensus_peers.cc:597] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72e86c671bb747c785d8f676ed3b8315. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:44.954315 11303 raft_consensus.cc:1275] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:44.955876 12228 consensus_queue.cc:1048] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:44.955539 11233 raft_consensus.cc:1275] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:44.961509 12228 consensus_queue.cc:1048] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:44.979738 12142 raft_consensus.cc:2955] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:44.991776 11303 raft_consensus.cc:2955] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:44.999428 11230 raft_consensus.cc:2955] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:45.044874 11229 consensus_queue.cc:237] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:45.091256 11303 raft_consensus.cc:1275] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:45.092895 12297 consensus_queue.cc:1048] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:45.095140 11379 raft_consensus.cc:1275] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260321 02:58:45.097316 11201 consensus_peers.cc:597] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d42ec1d798814a50bb535a7012a76eb4. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:45.100131 12297 consensus_queue.cc:1048] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:45.101477 11134 catalog_manager.cc:5671] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:45.126250 12297 raft_consensus.cc:2955] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:45.133011 11307 raft_consensus.cc:2955] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:45.198875 11132 catalog_manager.cc:5671] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:45.206449 11379 consensus_queue.cc:237] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
W20260321 02:58:45.252032 11350 consensus_peers.cc:597] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c6a0a98407b42d18543015ce289b654. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:45.253494 11307 raft_consensus.cc:1275] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:45.256091 12142 consensus_queue.cc:1048] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:45.262156 11229 raft_consensus.cc:1275] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:45.261318 11379 raft_consensus.cc:2955] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:45.266076 12142 consensus_queue.cc:1048] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:45.313241 12275 raft_consensus.cc:2955] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:45.322140 11306 raft_consensus.cc:2955] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:45.323819 11230 raft_consensus.cc:2955] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:45.342919 11134 catalog_manager.cc:5671] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: config changed from index -1 to 2, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:58:45.372843 12322 ts_tablet_manager.cc:933] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:45.378368 12324 ts_tablet_manager.cc:933] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:45.383581 12322 tablet_copy_client.cc:323] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.3:43533
I20260321 02:58:45.385360 11395 tablet_copy_service.cc:140] P fef7f8cc55b043039131bd9b2229f0a5: Received BeginTabletCopySession request for tablet 72e86c671bb747c785d8f676ed3b8315 from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:45634)
I20260321 02:58:45.385962 11395 tablet_copy_service.cc:161] P fef7f8cc55b043039131bd9b2229f0a5: Beginning new tablet copy session on tablet 72e86c671bb747c785d8f676ed3b8315 from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:45634: session id = 3d8d3176d6ec407ea2a965556c2e2c56-72e86c671bb747c785d8f676ed3b8315
I20260321 02:58:45.393859 12324 tablet_copy_client.cc:323] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.2:34267
I20260321 02:58:45.396198 11321 tablet_copy_service.cc:140] P 83b12b883a7f4129a87dadf5a78f08da: Received BeginTabletCopySession request for tablet 76097e48b77445aaa1ffe490b8bb41cf from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:49786)
I20260321 02:58:45.396806 11321 tablet_copy_service.cc:161] P 83b12b883a7f4129a87dadf5a78f08da: Beginning new tablet copy session on tablet 76097e48b77445aaa1ffe490b8bb41cf from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:49786: session id = 3d8d3176d6ec407ea2a965556c2e2c56-76097e48b77445aaa1ffe490b8bb41cf
I20260321 02:58:45.399412 11395 tablet_copy_source_session.cc:215] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:58:45.402922 12322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72e86c671bb747c785d8f676ed3b8315. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:45.408814 11321 tablet_copy_source_session.cc:215] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:58:45.416783 12324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76097e48b77445aaa1ffe490b8bb41cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:45.431156 12322 tablet_copy_client.cc:806] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
I20260321 02:58:45.434966 12322 tablet_copy_client.cc:670] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:58:45.440543 12324 tablet_copy_client.cc:806] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
I20260321 02:58:45.441138 12324 tablet_copy_client.cc:670] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:58:45.441550 12322 tablet_copy_client.cc:538] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 02:58:45.450492 12324 tablet_copy_client.cc:538] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 02:58:45.455106 12322 tablet_bootstrap.cc:492] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap starting.
I20260321 02:58:45.467773 12324 tablet_bootstrap.cc:492] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap starting.
I20260321 02:58:45.499591 12322 tablet_bootstrap.cc:492] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 02:58:45.500532 12322 tablet_bootstrap.cc:492] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap complete.
I20260321 02:58:45.501219 12322 ts_tablet_manager.cc:1403] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent bootstrapping tablet: real 0.046s user 0.013s sys 0.028s
I20260321 02:58:45.504477 12322 raft_consensus.cc:359] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:45.505667 12322 raft_consensus.cc:740] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Initialized, Role: LEARNER
I20260321 02:58:45.507071 12322 consensus_queue.cc:260] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:45.521180 12324 tablet_bootstrap.cc:492] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 02:58:45.526074 12324 tablet_bootstrap.cc:492] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap complete.
I20260321 02:58:45.526901 12324 ts_tablet_manager.cc:1403] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent bootstrapping tablet: real 0.065s user 0.023s sys 0.009s
I20260321 02:58:45.532181 12324 raft_consensus.cc:359] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:45.534478 12324 raft_consensus.cc:740] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Initialized, Role: LEARNER
I20260321 02:58:45.535182 12324 consensus_queue.cc:260] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:45.538586 12322 ts_tablet_manager.cc:1434] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent starting tablet: real 0.037s user 0.014s sys 0.006s
I20260321 02:58:45.540376 11395 tablet_copy_service.cc:342] P fef7f8cc55b043039131bd9b2229f0a5: Request end of tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-72e86c671bb747c785d8f676ed3b8315 received from {username='slave'} at 127.0.0.1:45634
I20260321 02:58:45.540935 11395 tablet_copy_service.cc:434] P fef7f8cc55b043039131bd9b2229f0a5: ending tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-72e86c671bb747c785d8f676ed3b8315 on tablet 72e86c671bb747c785d8f676ed3b8315 with peer 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:45.546169 12324 ts_tablet_manager.cc:1434] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent starting tablet: real 0.017s user 0.003s sys 0.003s
I20260321 02:58:45.548749 11321 tablet_copy_service.cc:342] P 83b12b883a7f4129a87dadf5a78f08da: Request end of tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-76097e48b77445aaa1ffe490b8bb41cf received from {username='slave'} at 127.0.0.1:49786
I20260321 02:58:45.549244 11321 tablet_copy_service.cc:434] P 83b12b883a7f4129a87dadf5a78f08da: ending tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-76097e48b77445aaa1ffe490b8bb41cf on tablet 76097e48b77445aaa1ffe490b8bb41cf with peer 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:45.638756 12324 ts_tablet_manager.cc:933] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717)
I20260321 02:58:45.642933 12324 tablet_copy_client.cc:323] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.1:36717
I20260321 02:58:45.644562 11246 tablet_copy_service.cc:140] P 0d7efc590a1041a9bd4cd4a971332dfb: Received BeginTabletCopySession request for tablet d42ec1d798814a50bb535a7012a76eb4 from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:59202)
I20260321 02:58:45.645154 11246 tablet_copy_service.cc:161] P 0d7efc590a1041a9bd4cd4a971332dfb: Beginning new tablet copy session on tablet d42ec1d798814a50bb535a7012a76eb4 from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:59202: session id = 3d8d3176d6ec407ea2a965556c2e2c56-d42ec1d798814a50bb535a7012a76eb4
I20260321 02:58:45.652184 12322 ts_tablet_manager.cc:933] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:45.669373 12322 tablet_copy_client.cc:323] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.3:43533
I20260321 02:58:45.671166 11395 tablet_copy_service.cc:140] P fef7f8cc55b043039131bd9b2229f0a5: Received BeginTabletCopySession request for tablet 7c6a0a98407b42d18543015ce289b654 from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:45634)
I20260321 02:58:45.671726 11395 tablet_copy_service.cc:161] P fef7f8cc55b043039131bd9b2229f0a5: Beginning new tablet copy session on tablet 7c6a0a98407b42d18543015ce289b654 from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:45634: session id = 3d8d3176d6ec407ea2a965556c2e2c56-7c6a0a98407b42d18543015ce289b654
I20260321 02:58:45.676331 11246 tablet_copy_source_session.cc:215] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:58:45.678403 11395 tablet_copy_source_session.cc:215] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:58:45.681689 12322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c6a0a98407b42d18543015ce289b654. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:45.685639 12324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d42ec1d798814a50bb535a7012a76eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:58:45.705938 12324 tablet_copy_client.cc:806] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
I20260321 02:58:45.706944 12324 tablet_copy_client.cc:670] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:58:45.715473 12324 tablet_copy_client.cc:538] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 02:58:45.730291 12322 tablet_copy_client.cc:806] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
I20260321 02:58:45.731207 12322 tablet_copy_client.cc:670] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:58:45.736208 12322 tablet_copy_client.cc:538] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260321 02:58:45.748049 12322 tablet_bootstrap.cc:492] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap starting.
I20260321 02:58:45.753275 12324 tablet_bootstrap.cc:492] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap starting.
I20260321 02:58:45.786504 12322 tablet_bootstrap.cc:492] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 02:58:45.787598 12322 tablet_bootstrap.cc:492] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap complete.
I20260321 02:58:45.788395 12322 ts_tablet_manager.cc:1403] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent bootstrapping tablet: real 0.041s user 0.027s sys 0.007s
I20260321 02:58:45.792140 12324 tablet_bootstrap.cc:492] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260321 02:58:45.799100 12324 tablet_bootstrap.cc:492] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56: Bootstrap complete.
I20260321 02:58:45.799984 12324 ts_tablet_manager.cc:1403] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent bootstrapping tablet: real 0.047s user 0.021s sys 0.007s
I20260321 02:58:45.796644 12322 raft_consensus.cc:359] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:45.800709 12322 raft_consensus.cc:740] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Initialized, Role: LEARNER
I20260321 02:58:45.801438 12322 consensus_queue.cc:260] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:45.804388 12324 raft_consensus.cc:359] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:45.805280 12322 ts_tablet_manager.cc:1434] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent starting tablet: real 0.015s user 0.004s sys 0.003s
I20260321 02:58:45.805294 12324 raft_consensus.cc:740] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Initialized, Role: LEARNER
I20260321 02:58:45.807152 12324 consensus_queue.cc:260] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:45.812220 11395 tablet_copy_service.cc:342] P fef7f8cc55b043039131bd9b2229f0a5: Request end of tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-7c6a0a98407b42d18543015ce289b654 received from {username='slave'} at 127.0.0.1:45634
I20260321 02:58:45.812705 12324 ts_tablet_manager.cc:1434] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260321 02:58:45.812726 11395 tablet_copy_service.cc:434] P fef7f8cc55b043039131bd9b2229f0a5: ending tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-7c6a0a98407b42d18543015ce289b654 on tablet 7c6a0a98407b42d18543015ce289b654 with peer 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:45.814685 11246 tablet_copy_service.cc:342] P 0d7efc590a1041a9bd4cd4a971332dfb: Request end of tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-d42ec1d798814a50bb535a7012a76eb4 received from {username='slave'} at 127.0.0.1:59202
I20260321 02:58:45.815092 11246 tablet_copy_service.cc:434] P 0d7efc590a1041a9bd4cd4a971332dfb: ending tablet copy session 3d8d3176d6ec407ea2a965556c2e2c56-d42ec1d798814a50bb535a7012a76eb4 on tablet d42ec1d798814a50bb535a7012a76eb4 with peer 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:45.852707 11979 raft_consensus.cc:1217] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260321 02:58:45.998842 11979 raft_consensus.cc:1217] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260321 02:58:46.053059 11979 raft_consensus.cc:1217] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260321 02:58:46.243906 11979 raft_consensus.cc:1217] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260321 02:58:46.295089 12228 raft_consensus.cc:1064] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5: attempting to promote NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 to VOTER
I20260321 02:58:46.296975 12228 consensus_queue.cc:237] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:46.305058 11979 raft_consensus.cc:1275] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:46.305013 11229 raft_consensus.cc:1275] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:46.305058 11306 raft_consensus.cc:1275] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:46.306851 12142 consensus_queue.cc:1048] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:46.307847 12334 consensus_queue.cc:1048] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:46.316361 12321 consensus_queue.cc:1048] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:46.400362 12321 raft_consensus.cc:2955] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:46.405649 12281 raft_consensus.cc:1064] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da: attempting to promote NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 to VOTER
I20260321 02:58:46.408870 12281 consensus_queue.cc:237] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:46.410254 11307 raft_consensus.cc:2955] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:46.424623 11979 raft_consensus.cc:2955] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:46.416916 11229 raft_consensus.cc:2955] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:46.489096 11233 raft_consensus.cc:1275] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:46.502393 11377 raft_consensus.cc:1275] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:46.515336 11979 raft_consensus.cc:1275] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:46.538751 12228 raft_consensus.cc:1064] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5: attempting to promote NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 to VOTER
I20260321 02:58:46.537299 11132 catalog_manager.cc:5671] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:46.551424 12283 consensus_queue.cc:1048] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:46.553059 12281 consensus_queue.cc:1048] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:46.574033 12228 consensus_queue.cc:237] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:58:46.650990 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 102100us
W20260321 02:58:46.623229 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 90256.1us
I20260321 02:58:46.669916 12281 consensus_queue.cc:1048] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:46.718225 11978 raft_consensus.cc:1275] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:46.736903 12334 consensus_queue.cc:1048] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:46.764045 11307 raft_consensus.cc:1275] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:46.813495 12349 raft_consensus.cc:1064] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb: attempting to promote NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 to VOTER
I20260321 02:58:46.817306 12349 consensus_queue.cc:237] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:46.818039 11235 raft_consensus.cc:1275] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:46.841804 12228 consensus_queue.cc:1048] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:46.850534 12344 raft_consensus.cc:2955] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:46.856720 12321 consensus_queue.cc:1048] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:58:46.868294 11978 raft_consensus.cc:2955] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:46.885433 11230 raft_consensus.cc:2955] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:46.891849 11382 raft_consensus.cc:1275] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:46.891920 11979 raft_consensus.cc:1275] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 LEARNER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:46.893399 12349 consensus_queue.cc:1048] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:46.894526 12315 consensus_queue.cc:1048] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260321 02:58:46.898492 11302 raft_consensus.cc:1275] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260321 02:58:46.899991 12315 consensus_queue.cc:1048] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:58:46.903116 11382 raft_consensus.cc:2955] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:46.929886 12228 raft_consensus.cc:2955] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:46.935920 12349 raft_consensus.cc:2955] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:46.959645 11382 consensus_queue.cc:237] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:46.959306 11977 raft_consensus.cc:2955] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:46.967051 11132 catalog_manager.cc:5671] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:46.975175 11309 raft_consensus.cc:2955] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:46.981913 11377 raft_consensus.cc:2955] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:46.994422 11302 consensus_queue.cc:237] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:47.003907 11977 raft_consensus.cc:1275] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:47.004467 11231 raft_consensus.cc:1275] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:47.003039 11230 raft_consensus.cc:2955] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:47.005486 12346 consensus_queue.cc:1048] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:47.006513 12345 consensus_queue.cc:1048] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:47.011519 11134 catalog_manager.cc:5671] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: true } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:47.015794 12344 raft_consensus.cc:2955] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:47.025470 11302 raft_consensus.cc:2955] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:47.028259 11231 raft_consensus.cc:2955] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:47.051303 11119 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 76097e48b77445aaa1ffe490b8bb41cf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260321 02:58:47.052742 11978 raft_consensus.cc:1275] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:47.054672 12275 consensus_queue.cc:1048] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:47.066009 12142 raft_consensus.cc:2955] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:47.067783 11978 raft_consensus.cc:2955] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:47.072271 11133 catalog_manager.cc:5671] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:47.083863 11233 consensus_queue.cc:237] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:47.089951 11306 raft_consensus.cc:1275] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:47.089389 11130 catalog_manager.cc:5671] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:47.089846 11979 raft_consensus.cc:1275] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:47.092934 11979 raft_consensus.cc:2955] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } attrs { replace: true } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:47.105146 12315 consensus_queue.cc:1048] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:47.091776 11977 raft_consensus.cc:2955] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:47.107152 12315 consensus_queue.cc:1048] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:47.121661 11217 tablet_service.cc:1558] Processing DeleteTablet for tablet 72e86c671bb747c785d8f676ed3b8315 with delete_type TABLET_DATA_TOMBSTONED (TS 0d7efc590a1041a9bd4cd4a971332dfb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58196
I20260321 02:58:47.128165 12317 raft_consensus.cc:2955] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:47.132822 11976 raft_consensus.cc:2955] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:47.144750 11306 raft_consensus.cc:1275] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:47.157555 11309 raft_consensus.cc:2955] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:47.185002 12367 tablet_replica.cc:333] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb: stopping tablet replica
I20260321 02:58:47.221285 12367 raft_consensus.cc:2243] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:47.222327 12367 raft_consensus.cc:2272] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:47.222779 11379 consensus_queue.cc:237] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:47.227306 12367 ts_tablet_manager.cc:1916] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 02:58:47.190068 11133 catalog_manager.cc:5671] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New cstate: current_term: 1 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:58:47.200007 11119 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 72e86c671bb747c785d8f676ed3b8315 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260321 02:58:47.231316 11119 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet d42ec1d798814a50bb535a7012a76eb4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260321 02:58:47.211728 12275 consensus_queue.cc:1048] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:47.255420 12367 ts_tablet_manager.cc:1929] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260321 02:58:47.256145 12367 log.cc:1199] T 72e86c671bb747c785d8f676ed3b8315 P 0d7efc590a1041a9bd4cd4a971332dfb: Deleting WAL directory at /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-0-root/wals/72e86c671bb747c785d8f676ed3b8315
I20260321 02:58:47.262290 11976 raft_consensus.cc:1275] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:47.263196 11130 catalog_manager.cc:5671] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: config changed from index 3 to 4, VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) evicted. New cstate: current_term: 1 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:47.264900 12368 consensus_queue.cc:1048] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:58:47.271160 11119 catalog_manager.cc:5002] TS 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): tablet 72e86c671bb747c785d8f676ed3b8315 (table test-workload [id=07ae567e73b94faa9ae558018daa9713]) successfully deleted
W20260321 02:58:47.285228 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 57246.6us
I20260321 02:58:47.296743 11366 tablet_service.cc:1558] Processing DeleteTablet for tablet d42ec1d798814a50bb535a7012a76eb4 with delete_type TABLET_DATA_TOMBSTONED (TS fef7f8cc55b043039131bd9b2229f0a5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40932
I20260321 02:58:47.301030 11368 tablet_service.cc:1558] Processing DeleteTablet for tablet 76097e48b77445aaa1ffe490b8bb41cf with delete_type TABLET_DATA_TOMBSTONED (TS fef7f8cc55b043039131bd9b2229f0a5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40932
I20260321 02:58:47.313092 12142 raft_consensus.cc:2955] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:47.314318 11979 raft_consensus.cc:2955] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:47.322546 11119 catalog_manager.cc:5184] ChangeConfig:REMOVE_PEER RPC for tablet 7c6a0a98407b42d18543015ce289b654 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260321 02:58:47.327116 12369 tablet_replica.cc:333] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5: stopping tablet replica
I20260321 02:58:47.329494 12369 raft_consensus.cc:2243] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:47.330399 12369 raft_consensus.cc:2272] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:47.335439 11302 raft_consensus.cc:1275] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260321 02:58:47.337033 11306 raft_consensus.cc:2955] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:47.348196 12369 ts_tablet_manager.cc:1916] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 02:58:47.347841 11133 catalog_manager.cc:5671] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New cstate: current_term: 1 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:58:47.364387 11217 tablet_service.cc:1558] Processing DeleteTablet for tablet 7c6a0a98407b42d18543015ce289b654 with delete_type TABLET_DATA_TOMBSTONED (TS 0d7efc590a1041a9bd4cd4a971332dfb not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58196
I20260321 02:58:47.365443 12367 tablet_replica.cc:333] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb: stopping tablet replica
I20260321 02:58:47.366580 12367 raft_consensus.cc:2243] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:47.367383 12367 raft_consensus.cc:2272] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:47.371466 12367 ts_tablet_manager.cc:1916] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 02:58:47.387317 12369 ts_tablet_manager.cc:1929] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260321 02:58:47.387984 12369 log.cc:1199] T d42ec1d798814a50bb535a7012a76eb4 P fef7f8cc55b043039131bd9b2229f0a5: Deleting WAL directory at /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-2-root/wals/d42ec1d798814a50bb535a7012a76eb4
I20260321 02:58:47.390148 11119 catalog_manager.cc:5002] TS fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): tablet d42ec1d798814a50bb535a7012a76eb4 (table test-workload [id=07ae567e73b94faa9ae558018daa9713]) successfully deleted
I20260321 02:58:47.390240 12369 tablet_replica.cc:333] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5: stopping tablet replica
I20260321 02:58:47.391462 12369 raft_consensus.cc:2243] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260321 02:58:47.392546 12369 raft_consensus.cc:2272] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260321 02:58:47.400950 12275 consensus_queue.cc:1048] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:58:47.405934 12369 ts_tablet_manager.cc:1916] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260321 02:58:47.417541 12367 ts_tablet_manager.cc:1929] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260321 02:58:47.418133 12367 log.cc:1199] T 7c6a0a98407b42d18543015ce289b654 P 0d7efc590a1041a9bd4cd4a971332dfb: Deleting WAL directory at /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-0-root/wals/7c6a0a98407b42d18543015ce289b654
I20260321 02:58:47.420081 11119 catalog_manager.cc:5002] TS 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): tablet 7c6a0a98407b42d18543015ce289b654 (table test-workload [id=07ae567e73b94faa9ae558018daa9713]) successfully deleted
I20260321 02:58:47.425500 12369 ts_tablet_manager.cc:1929] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260321 02:58:47.426009 12369 log.cc:1199] T 76097e48b77445aaa1ffe490b8bb41cf P fef7f8cc55b043039131bd9b2229f0a5: Deleting WAL directory at /tmp/dist-test-tasksA5jWN/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1774061835490702-10068-0/minicluster-data/ts-2-root/wals/76097e48b77445aaa1ffe490b8bb41cf
I20260321 02:58:47.426789 11306 raft_consensus.cc:2955] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:58:47.427927 11119 catalog_manager.cc:5002] TS fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): tablet 76097e48b77445aaa1ffe490b8bb41cf (table test-workload [id=07ae567e73b94faa9ae558018daa9713]) successfully deleted
I20260321 02:58:48.659982 12372 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:48.660583 12372 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:48.662849 12372 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:48.727748 12372 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:48.728726 12372 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:48.946310 12374 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:48.947192 12374 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:48.947860 12370 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:48.956043 12370 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:48.959483 12370 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:48.951202 12375 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:48.968544 12375 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:48.951242 12373 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:48.971720 12373 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:48.972421 12375 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:48.973193 12375 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:48.974030 12375 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:48.961197 12376 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:48.976043 12389 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:48.976537 12376 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:48.979158 12376 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:48.962323 12377 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:48.963377 12378 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:48.963943 12371 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:48.964789 12381 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:48.949388 12374 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:48.966961 12382 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:48.974453 12373 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:48.960083 12380 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:48.976681 12389 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:48.731791 12372 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.002712 12372 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:48.980261 12377 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:48.996204 12371 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.001188 12380 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:48.991514 12378 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.019270 12372 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.026700 12373 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.027503 12373 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.028275 12372 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.031941 12386 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.035367 12389 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.036512 12375 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.037106 12375 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:49.037635 12375 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.039374 12388 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:49.021699 12374 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.039620 12375 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.032804 12386 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.048866 12375 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.049407 12375 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.050014 12386 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.009332 12381 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.005373 12382 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.055022 12374 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.044708 12388 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.066727 12378 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.071173 12378 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:49.073397 12391 raft_consensus.cc:493] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.063970 12375 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.064589 12373 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.069331 12377 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:49.081928 12377 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.082188 12385 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.082458 12376 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.082751 12373 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.074189 12391 raft_consensus.cc:515] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.080605 12375 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.083048 12376 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.079450 12384 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.087877 12379 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.087284 12375 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.088424 12382 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:49.083214 12373 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.078497 12378 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.089193 12388 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:49.096318 12383 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.096550 12381 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.097187 12393 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.082559 12377 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.088917 12386 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.101554 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:49.133723 12394 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.134830 12375 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.128485 12371 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.097635 12374 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:49.100244 12382 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:49.139498 12371 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.128297 12370 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:49.082759 12385 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.135425 12381 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.136559 12395 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.151525 12372 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.152791 12389 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.152081 12392 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:49.150873 12375 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.152987 12372 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.156562 12390 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.156821 12376 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.094950 12379 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.134598 12394 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.144129 12388 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.153482 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:49.157132 12390 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.152552 12370 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.153684 12375 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.164594 12391 leader_election.cc:290] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:49.143842 12386 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.147109 12374 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.101048 12384 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.146067 12382 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.153312 12389 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.148198 12371 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.158185 12380 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.153141 12396 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.165355 12376 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.158898 12397 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.142828 12393 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.177564 11978 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:49.154093 12381 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.153595 12392 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.185039 12371 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.185376 12382 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:49.185575 12371 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:49.185846 12392 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.168960 12391 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.181223 12387 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.181353 12375 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.182363 11980 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:49.185915 12382 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.122215 12383 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.167176 12388 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.180014 12373 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.179374 12378 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.170508 12374 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.190553 12377 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.186182 12371 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.185794 12381 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.173321 12395 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.199131 12379 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.200330 12389 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.202013 12370 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.178234 12396 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.203734 12388 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.188232 12376 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.207103 12373 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.192694 12391 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.222297 12386 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.223224 12371 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.223614 12374 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.196460 12382 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.215107 12379 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.231127 12384 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.233378 12395 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:49.220960 12385 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.227586 12376 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.206415 12397 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.224655 12398 raft_consensus.cc:493] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.242071 12396 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:49.224172 12373 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.225488 12399 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.210450 12387 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.250101 12275 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:49.235862 12379 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.252804 12275 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.282449 12380 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
W20260321 02:58:49.285032 11199 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.598s user 0.006s sys 0.005s
W20260321 02:58:49.285466 11199 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.667s user 0.007s sys 0.005s
I20260321 02:58:49.285804 12383 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.244943 12390 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:49.286290 12381 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:49.258304 12398 raft_consensus.cc:515] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.248217 12399 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.308902 12398 leader_election.cc:290] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:49.309959 12399 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:49.310932 12402 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:49.311064 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:49.311755 11976 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:49.312952 12275 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:49.302379 12377 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.295365 12396 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.294553 12394 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.287120 12393 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.300884 12383 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.329308 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.296631 12371 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:49.285115 12380 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.297940 12395 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:49.331166 12374 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.331705 12376 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.311514 12402 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.312536 12385 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.338290 11979 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:49.371873 11979 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:49.338941 12399 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.339246 12370 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.339530 12392 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:49.327771 12390 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.319722 12377 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.349017 12380 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.354235 12372 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.359308 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.333493 12394 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.365156 11201 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:58:49.332656 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:49.355463 12382 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.334331 12396 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.337736 12381 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:49.407843 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 80248.6us
I20260321 02:58:49.372330 12391 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:49.400560 12275 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.413653 12275 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.355798 12383 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.359653 12371 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.410387 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.344973 12395 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.359336 12374 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.361232 12376 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.401578 12404 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:49.398258 12389 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.398217 12401 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.403333 12373 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.407267 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:49.397210 12398 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.391191 12385 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.396232 12392 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.438098 12275 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:49.408425 12403 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.422871 11975 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:49.440653 12406 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.442269 12371 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.444939 12381 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.445097 11309 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:49.396787 12370 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.416169 12397 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.447463 12372 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.444887 12377 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.443780 12383 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.449862 12407 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.427800 12379 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.402863 12390 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.470288 12396 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.440143 12376 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.400869 12399 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.410737 12394 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.485648 12402 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:49.487263 12411 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.489125 12275 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.465523 12389 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.504594 12414 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.492789 12403 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.505870 12392 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.506269 12374 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:49.490530 12387 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.467391 12401 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.512337 12370 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.498364 12381 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.476236 12398 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.528224 12275 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.532390 12420 raft_consensus.cc:2804] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:49.498034 12372 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.536073 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.536854 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.529368 12383 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:49.522102 12409 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:49.473143 12404 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.546460 11306 raft_consensus.cc:2468] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:49.547192 12405 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:49.489833 12406 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.552217 12275 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:58:49.537612 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 103615us
I20260321 02:58:49.557688 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.566232 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:49.567979 11201 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:58:49.563889 11302 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:49.551906 12420 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.573448 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.574314 11306 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:49.565025 12383 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.572885 12420 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:49.507232 12407 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.581171 12275 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.528380 12413 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.587990 12406 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:49.588527 12275 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.593956 12403 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:49.595077 12421 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:49.536185 12414 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.599005 12414 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:49.576354 12389 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.520946 12411 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.615648 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:49.618916 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.619539 12275 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:49.622360 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:49.591883 12395 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.567967 12381 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.627209 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:49.627480 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:49.627406 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:49.628065 12392 raft_consensus.cc:2804] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:49.632359 11378 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:49.621564 12371 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.632448 12392 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.618072 12385 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.595927 12421 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.638480 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:49.643929 11382 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:49.638818 12401 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.640329 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:49.640429 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:49.649873 11231 raft_consensus.cc:2468] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:49.653435 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.653792 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.654250 11306 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:49.654911 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.655236 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.655467 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.655865 11303 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:49.656100 11309 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:49.646123 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:49.659432 11233 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:49.640417 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:49.636544 12392 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:49.660674 11228 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:49.668452 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:49.669324 11229 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:49.633513 12408 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:49.671213 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.671849 11302 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:49.570176 12409 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.675434 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.649119 12390 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:49.657874 12394 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.658116 12399 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.640576 12371 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.678565 11276 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:49.681384 11276 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:49.682159 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.682493 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:49.682770 12404 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:49.684114 11276 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:49.685851 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:49.690347 12410 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.691114 12410 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.687659 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:49.684502 12389 raft_consensus.cc:2804] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:49.693137 12374 raft_consensus.cc:2804] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:49.692921 12389 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.600992 12405 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.702477 12389 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:49.680881 12390 raft_consensus.cc:2804] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:49.613382 12413 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:49.692972 11383 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:49.689301 12383 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.681381 12398 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.709105 12411 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:49.697420 12374 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.719339 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:49.720909 12422 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:49.721122 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:49.721429 12381 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.703270 12372 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.716284 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:49.705766 12390 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.729599 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:49.732496 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.733173 11308 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:49.735091 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:49.723059 12374 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:49.702746 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:49.740010 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:49.740191 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:49.721566 12422 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.731016 12390 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:49.728353 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:49.746091 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.748921 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:49.747387 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.746444 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:49.729312 12417 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:49.680280 12408 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.745741 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:58:49.920534 12001 debug-util.cc:398] Leaking SignalData structure 0x7b0800248220 after lost signal to thread 10071
W20260321 02:58:49.809332 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 225112us
I20260321 02:58:49.777341 11378 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:49.900614 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.159010 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:49.981407 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:49.800881 12425 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:50.145987 12404 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:50.184465 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.184288 12404 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.185130 11378 raft_consensus.cc:2468] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 1.
I20260321 02:58:50.186199 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.186458 12404 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:49.794482 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.805282 12422 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:58:50.144158 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:58:49.749480 11233 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:50.145270 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:49.935752 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.871644 12420 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.850498 12371 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.869009 12405 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:49.850885 12413 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.824504 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:50.144654 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:49.805275 12410 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:49.775286 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.154373 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.807384 12407 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:49.806918 12409 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:49.802211 12421 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:50.165162 12417 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.162187 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.217406 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.218278 11302 raft_consensus.cc:2468] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 1.
I20260321 02:58:49.767298 12406 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:50.166829 12321 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:50.166339 12390 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:50.166977 12423 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:50.150853 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:58:50.166862 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 160078us
I20260321 02:58:49.940852 11385 consensus_queue.cc:237] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:50.196775 12389 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:49.804673 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:49.802297 12412 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.744004 12419 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.766851 11308 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:50.050550 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.233911 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.170854 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:49.805577 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.181612 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.181977 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:50.181667 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:50.235020 11228 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:50.183552 11235 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:50.184219 12425 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:50.185209 12429 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:49.934635 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.183372 11304 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:50.187775 11377 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:50.150358 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.194758 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.200737 11232 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:50.174798 12411 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:50.212435 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.214269 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.831033 12275 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:50.222549 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:58:50.220821 12001 debug-util.cc:398] Leaking SignalData structure 0x7b080009a920 after lost signal to thread 11168
I20260321 02:58:50.234012 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:49.742671 11227 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:50.222576 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.236421 11306 raft_consensus.cc:2393] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 2: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:50.261356 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.261979 11234 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:50.239820 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.263118 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.263504 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.264063 11302 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:50.264065 12411 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:50.266700 12411 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:50.267148 12408 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:50.239540 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.268252 11200 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.242101 11307 raft_consensus.cc:2393] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 2: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:50.241982 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.269085 11304 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:50.224442 12423 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.243975 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.272521 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.273051 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.273234 11231 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:50.273664 12423 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:50.245523 12425 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:50.274250 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.268187 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:50.275290 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.244920 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.275911 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.248940 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.276506 11231 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:50.248940 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.248126 11310 raft_consensus.cc:2468] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 1.
I20260321 02:58:50.247624 12275 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.253451 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.252630 12429 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:50.254370 12389 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:50.227123 12321 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.287808 11979 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:50.220963 12374 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:50.260001 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.260660 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.261950 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.290695 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.291076 12420 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:50.291285 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.291528 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.288239 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:50.235260 11236 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
I20260321 02:58:50.271037 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.240491 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.277262 11382 raft_consensus.cc:2468] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 1.
I20260321 02:58:50.293798 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.294874 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.295738 11979 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:58:50.273432 12392 consensus_meta.cc:287] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.599s user 0.008s sys 0.008s
I20260321 02:58:50.268419 11376 raft_consensus.cc:2468] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 1.
I20260321 02:58:50.222054 12406 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.279259 11200 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.280925 11380 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 1.
I20260321 02:58:50.280813 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.282056 12429 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:50.282838 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.289359 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:50.240767 12419 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.292586 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.319056 11200 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.319322 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.295826 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.319893 11200 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.320348 11382 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:50.253656 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.277951 11379 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:50.325507 11200 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.293201 11232 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:50.298589 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:50.295579 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:58:50.299144 12390 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:50.299602 11311 raft_consensus.cc:1240] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:50.331754 12370 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.332175 12370 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.332127 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:58:50.332669 12370 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.332912 11979 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:58:50.306803 12392 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:50.335769 12392 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:50.336350 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:58:50.300305 11942 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.338119 11200 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.339043 11200 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.340253 12422 raft_consensus.cc:2804] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:50.340656 12422 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.341032 12422 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:50.341962 11200 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.287887 12321 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:50.301693 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:50.302234 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.302917 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.302971 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.344631 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.346135 11229 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:50.303618 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:50.303730 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.305855 11304 raft_consensus.cc:1240] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:50.292503 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.306969 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.349643 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.350108 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.350626 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.279203 12379 raft_consensus.cc:2804] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260321 02:58:50.293710 12001 debug-util.cc:398] Leaking SignalData structure 0x7b08002f15c0 after lost signal to thread 11260
W20260321 02:58:50.356024 12001 debug-util.cc:398] Leaking SignalData structure 0x7b08002ab420 after lost signal to thread 11335
I20260321 02:58:50.350509 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:58:50.356874 12001 debug-util.cc:398] Leaking SignalData structure 0x7b080031e4c0 after lost signal to thread 11409
I20260321 02:58:50.357439 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.312125 11309 raft_consensus.cc:2393] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 2: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:50.312700 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.358287 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.240226 12412 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.313911 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.315024 12275 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:50.366000 12370 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.316982 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.366746 11230 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:50.317560 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:58:50.318338 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.322921 11377 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:50.368901 12406 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:50.369094 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.352195 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.369542 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.317649 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.327100 12413 raft_consensus.cc:2804] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.369735 11380 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:50.371058 12413 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.371537 12413 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.371809 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.328152 12408 raft_consensus.cc:2804] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.327280 12409 raft_consensus.cc:2804] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:50.324448 11305 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:50.373505 12409 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.373975 12409 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:50.329080 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.329679 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.330821 12374 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:50.343401 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.347894 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.352128 11200 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.354063 12420 raft_consensus.cc:2804] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.353013 12379 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.358801 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.381706 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:58:50.357944 12001 debug-util.cc:398] Leaking SignalData structure 0x7b0800102ac0 after lost signal to thread 12004
I20260321 02:58:50.360330 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.360759 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.360939 11306 raft_consensus.cc:1240] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:50.361176 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.361596 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.361454 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.362149 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.362964 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.344964 12392 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.316186 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.366825 11384 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:50.383052 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.367571 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.367842 12412 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:50.358451 11233 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:50.323854 12419 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:50.370342 11377 raft_consensus.cc:2468] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 1.
I20260321 02:58:50.404297 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.405388 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.405961 11377 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:50.407271 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.380131 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.409574 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:50.380514 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.373142 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.373119 12408 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.413347 12408 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.383435 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.374125 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.382907 12379 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.400429 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.400645 12417 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.400780 12370 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:58:50.379719 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 77808.3us
I20260321 02:58:50.401650 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.439435 11942 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.440263 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.395869 11228 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:50.409390 12422 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.379142 11200 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.442822 12408 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:50.443493 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 64425.7us
I20260321 02:58:50.401885 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.444111 11200 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.381599 12381 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.445117 12381 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.445541 12381 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.446029 11200 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.416098 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.381707 12420 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.450332 12420 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.419763 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.421919 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.424257 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:58:50.425112 12430 log.cc:927] Time spent T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20260321 02:58:50.426136 11309 raft_consensus.cc:1275] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260321 02:58:50.426348 12409 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.429411 12413 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.418253 12431 raft_consensus.cc:2804] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.441143 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:50.441418 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:50.402659 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:50.401180 12392 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.459395 11275 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.460899 12275 consensus_queue.cc:1059] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:58:50.457933 11200 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.462550 11275 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.463356 11275 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.465548 11235 raft_consensus.cc:2393] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 3: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:50.468116 12432 raft_consensus.cc:2804] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.468564 12432 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:50.468969 12432 raft_consensus.cc:3060] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.456408 12418 raft_consensus.cc:2804] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.462117 12392 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.457459 12417 raft_consensus.cc:2804] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.483491 12417 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.483996 12417 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.465586 12424 raft_consensus.cc:2804] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.492061 12424 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.492564 12424 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.469585 12411 raft_consensus.cc:3055] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:50.472999 12409 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.463735 12423 consensus_queue.cc:1059] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260321 02:58:50.462296 12431 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:50.465749 12401 raft_consensus.cc:2804] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.463562 12372 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.501403 12372 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.502048 12423 raft_consensus.cc:3055] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:50.502192 12372 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.502485 12423 raft_consensus.cc:740] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:50.481565 12413 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.474581 12418 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.457810 12370 raft_consensus.cc:2804] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.509900 12370 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.510453 12370 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.468628 12374 raft_consensus.cc:2804] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.514900 12374 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.515544 12374 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.515606 12420 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.494469 12411 raft_consensus.cc:740] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:50.517349 12411 consensus_queue.cc:260] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:50.518749 12411 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.500963 12401 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.501335 12431 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.509236 12418 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.515798 12424 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.499897 11200 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:58:50.521632 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 58324.5us
I20260321 02:58:50.521792 12401 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.523669 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.522646 11275 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.526324 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.526943 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.527733 11235 raft_consensus.cc:2393] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:50.528607 11228 raft_consensus.cc:2393] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:50.527087 12389 raft_consensus.cc:2804] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.530817 12425 consensus_queue.cc:1059] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:58:50.529837 12389 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.531603 12389 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.531665 12425 raft_consensus.cc:3055] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:50.532083 12425 raft_consensus.cc:740] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:50.532912 12425 consensus_queue.cc:260] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:50.534071 12425 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.533661 12379 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.503696 12392 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.529049 12408 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.547029 12392 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.520083 12423 consensus_queue.cc:260] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:50.525416 12413 raft_consensus.cc:2804] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.542938 12370 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.543560 12381 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.529899 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.541281 12379 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.551277 12413 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.551880 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.554364 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.555483 11275 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.562638 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.564664 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.565817 12433 raft_consensus.cc:2804] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.566207 12433 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.556219 12374 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.566618 12433 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.566885 11275 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.552508 12423 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.568228 11275 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.563746 11350 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:50.569437 11350 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:50.558805 12413 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.569756 12406 raft_consensus.cc:2749] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:50.570109 12321 raft_consensus.cc:2749] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:50.570775 11350 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:50.570188 12407 raft_consensus.cc:2804] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.571746 12390 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.574421 12407 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.574609 12390 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.575062 12390 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.574894 12407 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.571687 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.572714 11275 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.561743 12422 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.576968 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.575587 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.577009 12412 raft_consensus.cc:2749] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:50.599424 12374 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.585013 12434 raft_consensus.cc:2804] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.600885 12434 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.601343 12434 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.620406 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.610095 12417 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.623591 12417 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.625326 12424 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.643034 11228 raft_consensus.cc:1240] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3. Ops: []
I20260321 02:58:50.582443 12420 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.657352 12370 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.658138 11942 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:50.659134 12389 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.652117 12381 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.664672 12389 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.665175 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.666057 11230 raft_consensus.cc:1240] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:50.666126 11235 raft_consensus.cc:1240] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:50.668115 11236 raft_consensus.cc:1240] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
W20260321 02:58:50.668226 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 87311us
I20260321 02:58:50.670637 11275 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:50.670661 11231 raft_consensus.cc:1240] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:50.662231 12431 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:50.680485 12431 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:50.673288 11227 raft_consensus.cc:1240] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:50.681667 12435 raft_consensus.cc:2749] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:50.688448 11228 raft_consensus.cc:1240] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3. Ops: []
I20260321 02:58:50.663935 12432 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:50.699676 11976 raft_consensus.cc:1240] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:50.700752 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.701957 12420 consensus_queue.cc:1059] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:58:50.702706 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.671229 12418 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.708855 11228 raft_consensus.cc:1240] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:50.704957 12370 raft_consensus.cc:3055] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:50.706110 12431 raft_consensus.cc:2804] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.706749 12432 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:50.717811 11231 raft_consensus.cc:1240] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:50.712749 12370 raft_consensus.cc:740] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:50.713951 12431 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:50.721410 11231 raft_consensus.cc:1240] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
W20260321 02:58:50.715319 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 64131.8us
I20260321 02:58:50.727351 12401 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.736191 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.727401 12372 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.730898 12413 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.732626 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.732969 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.741492 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.742584 11232 raft_consensus.cc:1240] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:50.742645 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.743328 11234 raft_consensus.cc:1240] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:50.743642 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.739626 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.725437 12431 raft_consensus.cc:3060] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.743827 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.745440 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.745837 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.735773 12370 consensus_queue.cc:260] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:50.747084 12370 raft_consensus.cc:3060] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.747255 11227 raft_consensus.cc:1240] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:50.748607 11230 raft_consensus.cc:1240] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:50.738528 11228 raft_consensus.cc:1240] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:50.745517 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.737892 12401 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.725275 12418 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.752717 11236 raft_consensus.cc:1240] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:50.753612 12413 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.754668 12433 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.725566 12407 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.757212 12407 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.760521 11942 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:58:50.763155 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:58:50.767222 12432 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.763926 12431 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:50.769121 12431 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.772604 12372 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.772897 12432 raft_consensus.cc:2749] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:50.764775 12390 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.765784 12434 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:50.774070 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 103778us
I20260321 02:58:50.774969 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.775458 11275 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.776319 11275 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.776708 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.777086 11275 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.777009 12372 raft_consensus.cc:2804] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.777912 12407 raft_consensus.cc:2804] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.778070 11275 leader_election.cc:304] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:50.778363 12407 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.778681 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.778853 12401 raft_consensus.cc:2749] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:50.778968 12407 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.779918 11275 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.780637 12401 raft_consensus.cc:2804] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:50.781000 12401 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.781348 12401 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:50.781513 11236 raft_consensus.cc:1240] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:50.784104 12433 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.778134 11942 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:50.785246 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.785883 12432 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.777738 12372 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.787097 11275 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.800078 12407 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.801100 11275 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.801965 11275 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.802177 12407 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.802846 12407 raft_consensus.cc:2804] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.803108 11275 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:50.803543 12389 raft_consensus.cc:2804] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.803979 12435 raft_consensus.cc:2804] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.804010 12389 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.804234 12433 raft_consensus.cc:2749] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:50.804595 12389 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.804492 12435 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.803218 12407 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.816125 12407 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.793428 12401 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.816879 11200 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:50.817692 12413 raft_consensus.cc:2749] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:50.818228 12401 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.818569 11200 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:50.797264 12432 raft_consensus.cc:2749] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:50.790673 12372 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.788003 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:50.835110 11976 raft_consensus.cc:1240] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:50.805737 11275 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.799609 11942 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:50.836829 12401 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:50.837280 11275 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.837385 12401 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.838240 11275 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:50.838790 12432 raft_consensus.cc:2804] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.839217 12432 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:50.839407 12401 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.778375 12439 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.805243 12434 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.840032 12401 raft_consensus.cc:2804] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.823632 12407 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.839715 12432 raft_consensus.cc:3060] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.824481 11974 raft_consensus.cc:1240] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:50.841974 12407 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.839923 12433 raft_consensus.cc:2804] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.842552 12407 consensus_queue.cc:1059] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:58:50.842885 12433 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.843266 12433 raft_consensus.cc:3060] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.805160 12435 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.837486 11942 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:50.804901 12390 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.840305 12437 raft_consensus.cc:2749] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:50.844097 12413 consensus_queue.cc:1059] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260321 02:58:50.843611 12407 raft_consensus.cc:3055] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:50.839282 11275 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:50.844520 12439 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.813004 12389 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.846318 12439 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.847936 12431 raft_consensus.cc:2804] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.848194 12437 raft_consensus.cc:2804] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.848469 12431 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:50.848559 12418 raft_consensus.cc:3055] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:50.848701 12437 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.844744 12401 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:50.850919 12401 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.851027 11942 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:50.851928 12438 raft_consensus.cc:2804] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.852137 11942 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:50.852412 12438 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:50.852864 12438 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.849998 11200 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:50.850046 12413 raft_consensus.cc:2749] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:50.847683 12407 raft_consensus.cc:740] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:58:50.849033 12431 raft_consensus.cc:3060] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.849131 12418 raft_consensus.cc:740] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:50.855144 11275 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:50.854777 12437 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.856029 12422 raft_consensus.cc:2749] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:50.857280 11200 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:50.858007 12418 consensus_queue.cc:260] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:50.860661 12407 consensus_queue.cc:260] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:50.864768 11275 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:50.865119 11200 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:50.864086 12390 raft_consensus.cc:2804] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.868127 12390 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:50.868597 12390 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.865916 12422 raft_consensus.cc:2749] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:50.869764 12422 raft_consensus.cc:2804] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.869838 12434 raft_consensus.cc:2804] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.870178 12422 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:50.870323 12434 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:50.870620 12422 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.870772 12434 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.872481 12433 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.874462 12433 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.873980 12418 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.880901 12407 raft_consensus.cc:3060] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.868610 11200 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:50.887266 11275 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:50.888680 12413 raft_consensus.cc:2749] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:50.889071 11942 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:50.889427 11275 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:50.897737 12374 raft_consensus.cc:2804] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.902271 12374 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:50.902690 12374 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.890779 12431 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:50.894699 12438 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:50.896329 12436 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:50.897023 12434 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.886229 12389 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.885994 12372 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.925724 12372 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.923470 12436 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.930548 12390 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.933424 12390 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.934755 12438 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.936719 12431 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.937263 12435 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:50.954757 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 65386.6us
I20260321 02:58:50.938772 12401 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.955838 12436 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.955863 12435 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.957557 12401 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.958366 11275 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:50.940105 11200 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:50.961552 11200 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:50.938802 12434 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.961817 12418 raft_consensus.cc:2749] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:50.910105 12433 raft_consensus.cc:2804] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.962643 12413 raft_consensus.cc:2749] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:50.940805 12374 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.963105 12433 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:50.964578 12433 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.965679 11275 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:50.966450 12442 raft_consensus.cc:2804] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.966847 12442 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:50.967262 12442 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.967170 11200 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:50.968231 12420 raft_consensus.cc:2749] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:50.944460 12432 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:50.961603 12401 raft_consensus.cc:2804] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.959621 12440 raft_consensus.cc:2749] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:50.969758 11275 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:50.923100 12422 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:50.960299 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 71321.2us
I20260321 02:58:50.944460 12437 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.979662 11275 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:50.958513 12431 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.982813 12432 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.983042 12374 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:50.983377 12390 raft_consensus.cc:2804] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.984732 12390 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:50.985177 12390 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.986768 12439 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:50.981695 11275 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:50.971510 12401 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:50.988922 11275 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:50.989375 12374 raft_consensus.cc:2804] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:50.990790 11200 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:50.992324 12374 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:50.992760 12374 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.995102 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.996143 12379 raft_consensus.cc:2749] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:50.996326 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.997798 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.995522 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:50.998989 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:51.000664 12401 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:50.980551 12434 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:51.003234 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.004660 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:51.001125 12407 raft_consensus.cc:2804] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:51.006666 12407 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:51.006795 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:51.007176 12407 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.007709 11308 raft_consensus.cc:1240] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.008697 12439 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:51.004964 12434 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:51.013069 12434 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.016755 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.018186 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:51.014871 11275 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.020884 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.020797 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:51.021060 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:51.024510 12435 raft_consensus.cc:2749] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.027092 11275 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.029167 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.031348 12407 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.031625 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.032084 12374 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.032603 12435 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:51.037997 12374 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:51.038730 11307 raft_consensus.cc:1240] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.035776 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.001412 11942 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.040892 12440 raft_consensus.cc:2749] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260321 02:58:51.041038 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 59219.5us
I20260321 02:58:51.042084 11942 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:51.043087 12440 raft_consensus.cc:2804] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:51.043453 12440 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:51.043807 12440 raft_consensus.cc:3060] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.033842 11275 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.045567 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:51.045886 12374 raft_consensus.cc:2749] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.038137 12407 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:51.047425 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:51.048566 11275 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.049299 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:51.049297 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.050876 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:51.050925 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.050966 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.053215 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.056774 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:51.057439 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:51.056988 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:51.056762 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.059235 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.061316 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.063113 11236 raft_consensus.cc:2393] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:51.065896 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.066179 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.068123 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.068395 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.068898 11236 raft_consensus.cc:2393] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:51.069348 11230 raft_consensus.cc:2393] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:51.075182 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.065291 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.077340 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.078114 11230 raft_consensus.cc:2393] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:51.078706 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:51.065161 12435 raft_consensus.cc:2749] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:58:51.079613 12425 consensus_queue.cc:1059] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:51.080442 11232 raft_consensus.cc:1240] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.080547 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.081501 11234 raft_consensus.cc:2393] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:51.081796 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.082475 11232 raft_consensus.cc:2393] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:51.084192 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:51.084676 11976 raft_consensus.cc:1240] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.085713 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:51.086807 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.086946 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.087857 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.088281 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:51.088346 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.088876 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.077123 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.089365 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.089679 11236 raft_consensus.cc:2393] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:51.090008 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.090426 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.091676 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.092833 12423 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:51.093433 12423 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:51.094300 12418 consensus_queue.cc:1059] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:58:51.095484 12418 raft_consensus.cc:3055] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.095515 12423 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:51.094794 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.107033 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.103269 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.108269 11229 raft_consensus.cc:2393] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:51.108749 11307 raft_consensus.cc:1240] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.112646 11307 raft_consensus.cc:1240] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.096340 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.114253 12401 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.096875 12437 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:51.097517 12411 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:51.095983 12418 raft_consensus.cc:740] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:51.099088 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:51.115995 12411 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.116473 12418 consensus_queue.cc:260] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:58:51.117120 11200 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:58:51.117910 12411 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:51.099704 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.082276 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:58:51.100981 11350 consensus_peers.cc:597] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5fc517dbbd646bfa0d307d4e42556ba. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:58:51.119069 12418 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.116504 12401 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:51.119393 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.078783 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:51.101266 11309 raft_consensus.cc:1240] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.098680 12422 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:51.101791 12425 raft_consensus.cc:3055] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:58:51.102700 11308 raft_consensus.cc:1240] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.120292 12425 raft_consensus.cc:740] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:51.120774 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:51.121248 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.121156 12425 consensus_queue.cc:260] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.056141 12407 raft_consensus.cc:2749] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.121953 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:51.122447 12425 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:51.123380 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.104530 11311 raft_consensus.cc:1240] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.124848 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:51.125416 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.126179 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.126757 11311 raft_consensus.cc:1240] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.126957 11385 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.126451 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.128094 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.128691 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.129976 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.130860 11230 raft_consensus.cc:2393] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 3: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:51.104453 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.133270 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.134030 11233 raft_consensus.cc:2393] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:51.134727 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:51.104730 11305 raft_consensus.cc:1240] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.105666 11304 raft_consensus.cc:1240] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.105494 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.136685 11235 raft_consensus.cc:2393] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:51.137363 11304 raft_consensus.cc:1240] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.137415 11305 raft_consensus.cc:1240] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.138567 11304 raft_consensus.cc:1240] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3. Ops: []
I20260321 02:58:51.138710 11305 raft_consensus.cc:1240] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.139154 12435 consensus_queue.cc:1059] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:51.105818 11310 raft_consensus.cc:1240] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.139658 11304 raft_consensus.cc:1240] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.103233 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.140434 12407 raft_consensus.cc:3055] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.140601 11231 raft_consensus.cc:2393] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:51.140846 12407 raft_consensus.cc:740] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:58:51.109903 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.101287 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.120877 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:51.103103 11302 raft_consensus.cc:1240] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.139810 11305 raft_consensus.cc:1240] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.142328 12407 consensus_queue.cc:260] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.103761 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.066191 12442 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.109954 11303 raft_consensus.cc:1240] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3. Ops: []
I20260321 02:58:51.113590 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.144309 12407 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.145232 12442 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:51.147332 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.148296 11311 raft_consensus.cc:1240] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.108345 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.131742 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.151675 12425 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:51.154112 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.155099 11230 raft_consensus.cc:2393] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:51.155607 11200 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.141289 11385 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
I20260321 02:58:51.156420 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.158653 12425 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.153496 11309 raft_consensus.cc:2468] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 2.
I20260321 02:58:51.161513 11200 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.162384 11311 raft_consensus.cc:1240] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.162806 11309 raft_consensus.cc:1240] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.164016 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:51.158828 12440 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:51.165714 12411 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:51.165835 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.166230 12411 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.161176 12420 raft_consensus.cc:2749] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.167150 12440 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:51.168167 12411 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:51.169049 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.169761 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.170841 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.163327 12390 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.172715 12425 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:51.174228 12381 raft_consensus.cc:2749] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.175539 12390 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:51.176630 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.177726 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.184185 11233 raft_consensus.cc:2468] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 2.
I20260321 02:58:51.184604 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.186492 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.187223 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.188427 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.189484 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.191179 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.241937 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:51.202873 11303 raft_consensus.cc:1240] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.227851 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.238924 11302 raft_consensus.cc:1240] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.191768 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.216344 12434 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.245332 12431 raft_consensus.cc:2749] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.246017 11309 raft_consensus.cc:1240] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.194264 11976 raft_consensus.cc:1240] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.239530 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:51.247887 12381 consensus_queue.cc:1059] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:58:51.248885 12417 raft_consensus.cc:3055] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.157785 11942 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.249341 12417 raft_consensus.cc:740] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
W20260321 02:58:51.250048 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 92714.3us
I20260321 02:58:51.250162 12417 consensus_queue.cc:260] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:51.251088 11942 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.252323 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.187866 11228 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 2.
I20260321 02:58:51.252641 12417 raft_consensus.cc:3060] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.254057 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.254777 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.255465 12425 consensus_queue.cc:1048] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:51.255599 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.256676 12431 raft_consensus.cc:2749] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.259678 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.262979 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:58:51.266436 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:51.200799 12433 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.197144 11200 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.267724 11306 raft_consensus.cc:1240] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
W20260321 02:58:51.268301 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 71231.4us
I20260321 02:58:51.269390 12425 consensus_queue.cc:1059] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:51.269114 11200 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.271232 12411 raft_consensus.cc:3055] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.271093 12418 raft_consensus.cc:2749] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.272173 12417 raft_consensus.cc:2749] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.272439 12433 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:51.248879 12434 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:51.273180 12431 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.271509 11200 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.271617 12411 raft_consensus.cc:740] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:51.279920 11200 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.279975 12417 raft_consensus.cc:2749] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.280830 12418 raft_consensus.cc:2749] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.281162 12431 raft_consensus.cc:2749] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.280613 12411 consensus_queue.cc:260] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.282022 11200 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.282298 11942 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.282791 12418 raft_consensus.cc:2749] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.283475 11942 leader_election.cc:304] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.284219 12440 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.285045 11942 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.285116 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.285575 12418 consensus_queue.cc:1059] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:51.286187 12423 consensus_queue.cc:1059] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:51.286736 12418 raft_consensus.cc:3055] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.287106 12418 raft_consensus.cc:740] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:51.287303 12405 raft_consensus.cc:3055] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.287648 12405 raft_consensus.cc:740] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:51.287885 12418 consensus_queue.cc:260] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.288496 12405 consensus_queue.cc:260] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.289129 12418 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
W20260321 02:58:51.289315 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:58:51.290403 12405 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.291898 12440 raft_consensus.cc:2749] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.294494 12411 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.301277 12425 consensus_queue.cc:1059] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:51.299281 11200 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.301759 12423 consensus_queue.cc:1059] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:51.303396 11236 raft_consensus.cc:2468] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 2.
I20260321 02:58:51.303452 11200 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.306571 11976 raft_consensus.cc:1240] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.258314 11308 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:51.314116 12442 consensus_queue.cc:1059] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:58:51.315325 12442 raft_consensus.cc:3055] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.315671 12442 raft_consensus.cc:740] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:58:51.316704 12417 raft_consensus.cc:2749] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.316540 12442 consensus_queue.cc:260] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:51.317581 12417 raft_consensus.cc:2749] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.318223 12442 raft_consensus.cc:3060] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.319922 12417 consensus_queue.cc:1059] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:51.321760 12381 raft_consensus.cc:3055] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.322127 12381 raft_consensus.cc:740] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:51.322202 11200 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.322979 12381 consensus_queue.cc:260] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.323518 11200 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.324402 12381 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.324841 11200 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.325523 12420 raft_consensus.cc:2749] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.326103 11200 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.326304 12420 raft_consensus.cc:2749] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.322978 12417 raft_consensus.cc:2749] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:58:51.312629 12405 consensus_queue.cc:1059] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260321 02:58:51.329100 12432 raft_consensus.cc:2749] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.329167 12440 raft_consensus.cc:2749] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260321 02:58:51.330500 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 45517us
I20260321 02:58:51.377161 11942 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.377396 12445 raft_consensus.cc:3055] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.377871 12445 raft_consensus.cc:740] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:51.377988 12440 raft_consensus.cc:2749] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.378750 11942 leader_election.cc:304] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.378973 12445 consensus_queue.cc:260] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.379542 12440 raft_consensus.cc:2749] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260321 02:58:51.380191 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 50531.8us
W20260321 02:58:51.381958 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 55918.5us
I20260321 02:58:51.382809 11200 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.381814 12444 consensus_queue.cc:1059] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:51.383028 12443 consensus_queue.cc:1059] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:51.384963 12418 raft_consensus.cc:2749] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.385120 12423 consensus_queue.cc:1059] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:51.386991 12423 consensus_queue.cc:1059] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:51.390322 11976 raft_consensus.cc:1240] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
W20260321 02:58:51.391028 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:58:51.392383 12417 raft_consensus.cc:2749] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.394032 12447 raft_consensus.cc:3055] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.394429 12447 raft_consensus.cc:740] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:51.394908 12417 consensus_queue.cc:1059] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:58:51.395370 12447 consensus_queue.cc:260] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.396617 12417 raft_consensus.cc:3055] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.396713 12447 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.397097 12417 raft_consensus.cc:740] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:51.397230 12425 raft_consensus.cc:3055] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.397678 12425 raft_consensus.cc:740] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:51.398097 12417 consensus_queue.cc:260] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:51.390164 11200 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.399444 12417 raft_consensus.cc:3060] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.399928 12449 raft_consensus.cc:3055] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.400327 12449 raft_consensus.cc:740] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:51.401180 12449 consensus_queue.cc:260] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.402479 12411 raft_consensus.cc:3055] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.402886 12411 raft_consensus.cc:740] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:51.403878 12411 consensus_queue.cc:260] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.412650 12411 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.413100 12381 raft_consensus.cc:2749] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.398439 12425 consensus_queue.cc:260] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.418550 12445 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.418957 12425 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.420094 12446 raft_consensus.cc:3055] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.417572 11200 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.420495 12446 raft_consensus.cc:740] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:51.421530 11200 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.422181 12381 raft_consensus.cc:2749] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:58:51.422031 12411 consensus_queue.cc:1059] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:51.421945 12446 consensus_queue.cc:260] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:51.428289 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 48493.8us
I20260321 02:58:51.429100 12446 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.429163 12422 raft_consensus.cc:2749] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.430562 11200 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.423022 12444 consensus_queue.cc:1059] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:51.424683 12405 raft_consensus.cc:3055] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.431432 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.437086 12443 raft_consensus.cc:3055] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.437530 12443 raft_consensus.cc:740] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:51.438565 12405 raft_consensus.cc:740] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:51.438455 12443 consensus_queue.cc:260] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.439613 12405 consensus_queue.cc:260] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.441053 11942 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:51.441342 12405 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.416162 12423 consensus_queue.cc:1059] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260321 02:58:51.442837 12450 raft_consensus.cc:3055] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.443262 12450 raft_consensus.cc:740] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:51.443372 12423 raft_consensus.cc:3055] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.443825 12423 raft_consensus.cc:740] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:51.444401 12450 consensus_queue.cc:260] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.444830 12423 consensus_queue.cc:260] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.445677 11200 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.402940 12449 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.445926 12450 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.448879 12440 raft_consensus.cc:2804] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:58:51.452026 12440 raft_consensus.cc:697] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:58:51.452245 12381 raft_consensus.cc:2749] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.445886 12423 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.453274 12440 consensus_queue.cc:237] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:51.454537 11200 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.457409 12381 raft_consensus.cc:2749] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260321 02:58:51.446192 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:58:51.451117 12417 raft_consensus.cc:2749] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.452371 11942 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.462177 12432 raft_consensus.cc:2749] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.466743 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:51.468185 12381 consensus_queue.cc:1059] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260321 02:58:51.476065 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 62073.2us
I20260321 02:58:51.483181 12443 raft_consensus.cc:3060] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.488163 12444 consensus_queue.cc:1059] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:51.488880 11349 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.488390 12445 consensus_queue.cc:1059] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:51.489408 12444 consensus_queue.cc:1059] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:51.491039 11133 catalog_manager.cc:5671] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 1 to 2, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4). New cstate: current_term: 2 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:58:51.492704 12417 raft_consensus.cc:3055] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.493654 12411 consensus_queue.cc:1059] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:51.494966 12450 raft_consensus.cc:3055] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.495918 12411 raft_consensus.cc:2749] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.496245 12417 raft_consensus.cc:740] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:51.497004 12444 raft_consensus.cc:3055] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:58:51.497092 12448 raft_consensus.cc:3055] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.494305 12446 consensus_queue.cc:1059] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:51.497577 12448 raft_consensus.cc:740] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:51.497449 12417 consensus_queue.cc:260] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.497416 12444 raft_consensus.cc:740] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:51.498844 12450 raft_consensus.cc:740] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:51.498546 12448 consensus_queue.cc:260] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.499691 12450 consensus_queue.cc:260] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.499855 12448 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.500977 12450 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.502336 12411 consensus_queue.cc:1059] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:58:51.504052 12411 raft_consensus.cc:3055] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.504416 12411 raft_consensus.cc:740] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:51.505234 12411 consensus_queue.cc:260] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:51.507035 12411 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.496521 12445 raft_consensus.cc:3055] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.507889 12417 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.508263 12445 raft_consensus.cc:740] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:51.509722 12445 consensus_queue.cc:260] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.510489 11349 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.511276 12445 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.511674 11349 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.500411 12447 raft_consensus.cc:3055] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.512965 12447 raft_consensus.cc:740] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:51.514072 12447 consensus_queue.cc:260] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.515249 12423 raft_consensus.cc:2749] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.515630 12447 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.516896 12446 raft_consensus.cc:2749] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260321 02:58:51.517930 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 53638.4us
I20260321 02:58:51.518967 12381 consensus_queue.cc:1059] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:51.520187 12422 raft_consensus.cc:3055] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.520573 12422 raft_consensus.cc:740] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:51.521428 12422 consensus_queue.cc:260] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.527526 11349 leader_election.cc:304] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:51.528946 12422 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.529446 12423 consensus_queue.cc:1059] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:58:51.500186 12444 consensus_queue.cc:260] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.530553 12423 raft_consensus.cc:2749] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.530783 12444 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:51.532676 12423 raft_consensus.cc:3055] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.538308 12423 raft_consensus.cc:740] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:51.539232 12423 consensus_queue.cc:260] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:51.540779 12423 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.528870 11349 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.542896 11349 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.544173 11349 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.545176 12446 raft_consensus.cc:2749] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.545902 12446 raft_consensus.cc:2749] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.545948 11349 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.547801 12446 raft_consensus.cc:2749] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.533516 11200 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.550009 12418 raft_consensus.cc:2749] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.550897 12418 consensus_queue.cc:1059] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:51.551708 12379 consensus_queue.cc:1059] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:51.552098 12418 raft_consensus.cc:3055] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.552618 12418 raft_consensus.cc:740] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:51.553375 12420 raft_consensus.cc:3055] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.553494 12418 consensus_queue.cc:260] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.554395 12420 raft_consensus.cc:740] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:51.555562 12418 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.556157 11349 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.557713 11349 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:51.559119 11349 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.558799 12420 consensus_queue.cc:260] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.559893 12450 raft_consensus.cc:2749] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.560133 12425 raft_consensus.cc:2749] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.560599 12450 raft_consensus.cc:2749] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.560963 12425 raft_consensus.cc:2749] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.561895 11349 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.562943 12447 raft_consensus.cc:2749] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.564388 11349 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:51.566049 11349 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:51.567049 12448 raft_consensus.cc:2749] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.567399 12443 raft_consensus.cc:2749] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.569406 12420 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.569952 11200 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.573804 12381 consensus_queue.cc:1059] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:51.567814 11349 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.575755 12417 raft_consensus.cc:3055] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.576037 12423 raft_consensus.cc:2749] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.575028 12381 raft_consensus.cc:2749] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.576196 12417 raft_consensus.cc:740] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:51.577287 11349 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.578027 12423 raft_consensus.cc:2749] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.578136 12417 consensus_queue.cc:260] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.579294 11349 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.579658 12417 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.580221 12423 raft_consensus.cc:2749] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.581977 12423 consensus_queue.cc:1059] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:51.583349 12447 raft_consensus.cc:3055] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.583712 12447 raft_consensus.cc:740] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
W20260321 02:58:51.532078 12451 consensus_queue.cc:1175] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" responder_term: 3 status { last_received { term: 2 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260321 02:58:51.587996 12447 consensus_queue.cc:260] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:51.590219 12447 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.590721 12381 consensus_queue.cc:1059] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:51.592331 11200 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.586890 11349 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.595211 11308 raft_consensus.cc:1240] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:51.595148 11942 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.595515 12422 raft_consensus.cc:2804] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:58:51.595682 12379 raft_consensus.cc:3055] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.596117 12379 raft_consensus.cc:740] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:51.596508 12451 raft_consensus.cc:2749] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.596807 11349 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.596983 12422 raft_consensus.cc:697] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:51.597213 12440 raft_consensus.cc:2749] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260321 02:58:51.597679 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:58:51.596992 12379 consensus_queue.cc:260] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.598543 11306 raft_consensus.cc:1240] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
W20260321 02:58:51.616201 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:58:51.614768 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:51.602866 11275 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.606689 11942 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.602844 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:51.625326 11942 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.624500 12432 raft_consensus.cc:2804] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:58:51.629513 11942 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.630246 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:51.633191 11275 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.633533 12422 consensus_queue.cc:237] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.634502 11275 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.635157 12431 raft_consensus.cc:2804] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:58:51.636840 11275 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.637842 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:51.602056 12445 raft_consensus.cc:2749] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.636269 12442 raft_consensus.cc:2749] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.637722 12434 raft_consensus.cc:2749] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.640028 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:51.640920 12440 raft_consensus.cc:2804] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:58:51.646207 12440 raft_consensus.cc:697] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:58:51.647110 12440 consensus_queue.cc:237] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:51.641602 12433 raft_consensus.cc:2749] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:58:51.628636 12432 raft_consensus.cc:697] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:58:51.642344 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:51.643112 12379 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.651142 12432 consensus_queue.cc:237] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:51.656646 11236 raft_consensus.cc:2468] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 2.
I20260321 02:58:51.669641 12434 raft_consensus.cc:2749] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.676155 12420 consensus_queue.cc:1059] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:51.678114 12418 raft_consensus.cc:3055] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.678642 12418 raft_consensus.cc:740] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:51.660305 11275 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.680742 11275 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.684396 12433 raft_consensus.cc:2749] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.685316 11275 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.683627 12442 raft_consensus.cc:2749] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.686952 12433 raft_consensus.cc:2749] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.691901 11275 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.693854 11275 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.683882 12431 raft_consensus.cc:697] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:58:51.702447 11275 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.701359 12442 raft_consensus.cc:2749] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.685668 12417 consensus_queue.cc:1059] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260321 02:58:51.702085 12433 raft_consensus.cc:2749] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.708431 11942 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.705780 12434 raft_consensus.cc:2749] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.705578 11275 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.685514 12418 consensus_queue.cc:260] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.711671 12434 raft_consensus.cc:2749] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.712694 12431 consensus_queue.cc:237] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:51.717507 11275 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.718883 11275 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.720741 11275 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.720602 12433 raft_consensus.cc:2749] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.723870 11275 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.725297 11275 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:51.720037 12432 raft_consensus.cc:2804] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:58:51.723549 12442 raft_consensus.cc:2749] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.730338 12433 raft_consensus.cc:2749] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.720073 12434 raft_consensus.cc:2749] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.729596 11275 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.733139 12407 raft_consensus.cc:2804] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:58:51.734157 12407 raft_consensus.cc:697] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 2 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:58:51.736370 12433 raft_consensus.cc:2749] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.736065 12407 consensus_queue.cc:237] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.736775 11275 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.737478 12442 raft_consensus.cc:2749] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260321 02:58:51.739300 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:58:51.744139 11130 catalog_manager.cc:5671] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 1 to 2, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 2 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:51.741336 12432 raft_consensus.cc:697] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:58:51.748584 12418 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.756623 12422 raft_consensus.cc:3055] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:58:51.757061 12422 raft_consensus.cc:740] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:51.758034 12422 consensus_queue.cc:260] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:51.760161 12422 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.767782 12432 consensus_queue.cc:237] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:51.755721 11275 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.776923 12442 raft_consensus.cc:2749] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.777386 11275 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.789659 11275 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.801676 12407 raft_consensus.cc:2749] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.801244 11275 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.807436 11132 catalog_manager.cc:5671] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 1 to 2, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 2 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:51.819367 11275 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.818885 12407 raft_consensus.cc:2749] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.820479 12442 raft_consensus.cc:2749] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.820861 11275 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.796737 12433 raft_consensus.cc:2749] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.822445 11275 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.824401 12433 raft_consensus.cc:2749] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.824868 11275 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.827345 11275 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.823119 12442 raft_consensus.cc:2749] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.828102 12407 raft_consensus.cc:2749] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.828056 12433 raft_consensus.cc:2749] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.829608 11275 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.831398 12407 raft_consensus.cc:2749] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.832379 11275 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.834550 12433 raft_consensus.cc:2749] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.837172 11275 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:51.838188 12407 raft_consensus.cc:2749] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.842301 11349 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.843327 12447 raft_consensus.cc:2749] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.843784 11236 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:58:51.844609 11236 raft_consensus.cc:606] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Received request to transfer leadership to TS 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:51.845042 11236 raft_consensus.cc:614] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:58:51.859608 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:51.861261 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:51.862743 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:51.864712 11308 raft_consensus.cc:1240] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 1. Current term is 2. Ops: [1.2-1.2]
I20260321 02:58:51.869601 11349 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.870826 12447 raft_consensus.cc:2749] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.873891 11349 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.875360 11349 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.875636 11133 catalog_manager.cc:5671] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 1 to 2, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4). New cstate: current_term: 2 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:58:51.877171 12445 raft_consensus.cc:2749] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.877171 12447 raft_consensus.cc:2749] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:51.886490 11133 catalog_manager.cc:5671] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 1 to 2, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4). New cstate: current_term: 2 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:58:51.888342 11133 catalog_manager.cc:5671] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 1 to 2, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4). New cstate: current_term: 2 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:58:51.890367 11376 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315"
dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
mode: GRACEFUL
new_leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
from {username='slave'} at 127.0.0.1:44010
I20260321 02:58:51.901400 11376 raft_consensus.cc:606] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Received request to transfer leadership to TS 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:51.901829 11376 raft_consensus.cc:614] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:58:51.901870 11133 catalog_manager.cc:5671] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 1 to 2, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4). New cstate: current_term: 2 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:58:51.905833 11236 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:58:51.906601 11236 raft_consensus.cc:606] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Received request to transfer leadership to TS 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:51.906924 11236 raft_consensus.cc:614] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:58:51.920214 11236 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "86406dbef163402197aa42a22bcc0101"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:58:51.920814 11236 raft_consensus.cc:606] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.921136 11236 raft_consensus.cc:614] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:58:51.921968 11376 raft_consensus.cc:3060] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:51.923696 11236 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:58:51.924412 11236 raft_consensus.cc:606] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.927058 11236 raft_consensus.cc:614] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:58:51.929935 11233 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:58:51.930543 11233 raft_consensus.cc:606] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:51.930903 11233 raft_consensus.cc:614] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:58:51.932516 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260321 02:58:51.932801 11376 raft_consensus.cc:1275] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260321 02:58:51.934120 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:58:51.934120 12432 consensus_queue.cc:1048] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:58:51.935547 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:58:51.946058 11308 raft_consensus.cc:1275] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260321 02:58:51.947728 12432 consensus_queue.cc:1048] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:58:52.041117 12447 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:52.041728 12447 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:52.045589 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:52.046450 11308 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:52.045542 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:52.047602 11349 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.048319 11976 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:52.048620 12445 raft_consensus.cc:2804] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:52.049153 12445 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:52.049620 12445 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.043849 12447 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:52.059203 11236 raft_consensus.cc:1275] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260321 02:58:52.060844 12454 consensus_queue.cc:1048] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:58:52.061185 12445 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:52.066895 12445 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:52.071851 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:58:52.074512 11976 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.079408 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:52.096083 11308 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.114178 11376 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:52.120045 11376 raft_consensus.cc:1275] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:58:52.122825 11308 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:52.123529 11976 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:52.124682 12417 consensus_queue.cc:1048] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:52.127386 11350 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.128170 12445 raft_consensus.cc:2804] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:52.128636 12445 raft_consensus.cc:697] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:52.129434 12445 consensus_queue.cc:237] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:52.149912 11383 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:52.152843 11233 raft_consensus.cc:1275] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260321 02:58:52.154727 12440 consensus_queue.cc:1048] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:58:52.160919 11383 raft_consensus.cc:1275] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260321 02:58:52.169168 12432 consensus_queue.cc:1048] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:58:52.172480 11311 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:52.182932 12477 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:52.183665 12477 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:52.185966 12477 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:52.186877 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:52.187709 11306 raft_consensus.cc:3060] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:52.187662 11308 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:52.190243 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:52.190954 11974 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:52.189714 11349 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.191213 11311 raft_consensus.cc:1275] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260321 02:58:52.192531 12477 raft_consensus.cc:2804] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:52.192971 12477 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:52.193390 12477 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.196125 11306 raft_consensus.cc:1275] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260321 02:58:52.203503 11383 raft_consensus.cc:1275] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:58:52.204516 12440 consensus_queue.cc:1048] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:58:52.213955 12440 consensus_queue.cc:1048] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:58:52.215905 12442 consensus_queue.cc:1048] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:52.216929 12477 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:52.237078 12477 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:52.238317 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:52.239025 11311 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.250185 11311 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:52.252287 11306 raft_consensus.cc:1275] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:58:52.270459 11233 raft_consensus.cc:1275] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260321 02:58:52.271363 11349 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.273240 11236 raft_consensus.cc:1275] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:58:52.278815 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:58:52.279381 11228 raft_consensus.cc:1275] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260321 02:58:52.280813 12417 consensus_queue.cc:1048] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:52.282882 11976 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.280997 12442 consensus_queue.cc:1048] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:58:52.294756 12432 consensus_queue.cc:1048] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:58:52.295596 12440 consensus_queue.cc:1048] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:58:52.292793 12445 raft_consensus.cc:2804] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:52.297587 12445 raft_consensus.cc:697] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:52.298405 12445 consensus_queue.cc:237] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:52.302670 11134 catalog_manager.cc:5671] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:52.305646 11383 raft_consensus.cc:3060] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:52.320160 11383 raft_consensus.cc:1275] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260321 02:58:52.326242 12487 consensus_queue.cc:1048] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:58:52.360756 12477 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:52.361330 12477 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:52.363260 12477 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:52.369828 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:52.370666 11974 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:52.372509 11350 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.375861 12477 raft_consensus.cc:2804] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:52.376600 12477 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:52.377151 12477 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.380609 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:52.381377 11309 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:52.408414 12417 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:52.408977 11976 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:52.409479 12417 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:52.421422 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:52.423386 12447 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:52.424252 12447 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:52.426834 12477 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:52.431703 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:58:52.432458 11974 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.432829 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:52.433697 11309 raft_consensus.cc:2468] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:52.436362 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:52.436569 12422 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:52.437114 12422 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.439157 12422 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:52.441087 11349 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.442324 11974 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:52.443471 11350 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.443931 11976 raft_consensus.cc:2393] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 2: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
I20260321 02:58:52.429510 12477 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:52.444794 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:52.445905 12477 raft_consensus.cc:2804] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:52.446058 11378 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:52.446419 12477 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:52.447012 12477 raft_consensus.cc:3060] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:52.448050 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:52.449052 11311 raft_consensus.cc:2393] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 2: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:52.458632 11200 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.459745 12445 raft_consensus.cc:2804] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:52.460253 12422 raft_consensus.cc:2804] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:52.460281 12445 raft_consensus.cc:697] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:52.460937 12422 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:52.461366 12422 raft_consensus.cc:3060] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:52.427102 12447 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:52.461458 12445 consensus_queue.cc:237] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:52.464133 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:52.469059 12417 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:52.479171 11309 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
W20260321 02:58:52.493085 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 52100.1us
I20260321 02:58:52.476349 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:52.501133 11976 raft_consensus.cc:2468] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:52.502415 11201 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:52.504087 12417 raft_consensus.cc:2804] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:52.504633 12417 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:52.505086 12417 raft_consensus.cc:3060] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.510196 12477 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:52.503041 11130 catalog_manager.cc:5671] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:52.518597 12422 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.522732 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:52.523500 11311 raft_consensus.cc:2393] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 2: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:52.524744 12477 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:52.528643 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:58:52.529651 11976 raft_consensus.cc:2393] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 2: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
I20260321 02:58:52.532047 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:52.540926 11309 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:52.545250 11311 raft_consensus.cc:2468] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:52.545782 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:52.546428 11349 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:52.548440 12445 raft_consensus.cc:2804] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:58:52.548938 12445 raft_consensus.cc:697] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 2 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:52.549767 12445 consensus_queue.cc:237] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:52.567619 11376 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:52.543007 12417 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:52.568334 12422 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:52.572297 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:52.573768 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:58:52.574401 11976 raft_consensus.cc:3060] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.578243 11200 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:52.584048 12418 raft_consensus.cc:2804] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:58:52.584583 12418 raft_consensus.cc:697] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:52.584731 11974 raft_consensus.cc:1275] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260321 02:58:52.585530 12418 consensus_queue.cc:237] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.586265 12477 consensus_queue.cc:1048] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:58:52.591212 12477 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:52.591941 12477 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.595853 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:52.596573 11233 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:52.595860 12477 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:52.591463 11976 raft_consensus.cc:2468] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:52.598126 11349 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.599447 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:52.599546 12477 raft_consensus.cc:2804] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:52.600256 11311 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:52.600451 12477 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:52.601917 11201 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:52.602500 12417 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:52.601074 12477 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.603399 12417 raft_consensus.cc:2804] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:52.604285 12417 raft_consensus.cc:697] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:52.605144 12417 consensus_queue.cc:237] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:52.600993 12445 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:52.608404 12445 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.611131 12445 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:52.613672 12477 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.611711 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:52.615674 11236 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:52.617249 11349 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.617728 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:52.618350 11309 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:52.617995 12477 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:52.619241 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:52.619776 11311 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.636608 11133 catalog_manager.cc:5671] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 1 to 2, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 2 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:52.643116 11311 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:52.629175 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:52.629215 11309 raft_consensus.cc:1275] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260321 02:58:52.647336 11236 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.649735 11349 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.650589 12497 raft_consensus.cc:2804] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:52.651093 12497 raft_consensus.cc:697] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:52.619411 12477 raft_consensus.cc:2804] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:52.652446 12477 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:52.652037 12497 consensus_queue.cc:237] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.652805 12477 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.650904 12447 consensus_queue.cc:1048] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:58:52.666949 11236 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:52.632741 11130 catalog_manager.cc:5671] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:52.679592 12445 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:52.680223 12445 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.682425 12445 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:52.683828 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:52.684561 11233 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:52.685874 11349 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.686867 12445 raft_consensus.cc:2804] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:52.687261 12445 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:52.687631 12445 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.691125 12477 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.693372 12477 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:52.695622 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:52.696303 11233 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.707307 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:52.707613 11233 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:52.708096 11309 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:52.707379 11132 catalog_manager.cc:5671] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:52.715168 11349 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.716209 12477 raft_consensus.cc:2804] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:52.716850 12477 raft_consensus.cc:697] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:52.717820 12477 consensus_queue.cc:237] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.725158 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:52.725703 11311 raft_consensus.cc:1275] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260321 02:58:52.725924 11309 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.737211 12445 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.739284 12445 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:52.744952 12497 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:52.745484 12445 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:52.745563 12497 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.746305 12445 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.748914 12445 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:52.749195 12497 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:52.755837 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:52.756119 12497 consensus_queue.cc:1048] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:58:52.757730 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:52.744901 11309 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:52.758360 11228 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:52.759585 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:52.760257 11230 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:52.756616 11236 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.763221 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:52.763850 11311 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:52.767238 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:52.767915 11309 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.768095 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:52.768874 11311 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:52.765842 11134 catalog_manager.cc:5671] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 2, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 2 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:52.774462 11134 catalog_manager.cc:5671] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:52.776510 11349 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.778105 11236 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:52.778230 12497 raft_consensus.cc:2804] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:52.778754 12497 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:52.779212 12497 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.786679 11974 raft_consensus.cc:1275] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260321 02:58:52.787169 11309 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:52.789319 12477 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:52.790278 12422 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:52.790771 12422 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.792908 12422 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:52.795373 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:52.795694 11349 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.790079 12477 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.803069 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:52.803085 11349 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.803916 11309 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:52.806213 11200 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:52.808228 12417 raft_consensus.cc:2804] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:52.809399 12417 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:52.807212 12447 raft_consensus.cc:2804] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:52.809885 12417 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.806433 12496 raft_consensus.cc:2804] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:52.810493 12496 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:52.810878 12496 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.814218 12497 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.815223 12422 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:52.815805 12422 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.816583 12497 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:52.818012 12422 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:52.818572 12434 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:52.819129 12434 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.821091 12434 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:52.822104 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:52.822757 11236 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:58:52.824194 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:52.824651 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:52.824883 11230 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.825311 11376 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:52.829900 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:52.830106 11275 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:52.830598 11302 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:52.832077 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:52.832538 12434 raft_consensus.cc:2804] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:58:52.833117 11230 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:52.832661 11309 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.834473 12497 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:52.835788 12460 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:52.803299 12445 consensus_queue.cc:1048] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260321 02:58:52.836293 12460 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.837762 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:52.838373 11236 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:52.839552 12460 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:52.833171 12434 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:52.836855 12417 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.842584 12417 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:52.840916 12434 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:52.834976 12497 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.843508 11200 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:52.847035 11349 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.847990 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:52.844184 12477 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:52.847908 11349 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.847836 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:52.848305 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:52.848925 11308 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:52.856762 11302 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:52.856775 11383 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:52.844898 12497 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:52.849929 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:52.860190 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:52.860486 11376 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:58:52.860811 11236 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:52.861104 12477 raft_consensus.cc:2804] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:52.861541 12477 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:52.861910 12477 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.852147 12445 raft_consensus.cc:2804] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:52.863942 12445 raft_consensus.cc:697] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:52.864333 11200 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.866276 12460 raft_consensus.cc:2804] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:52.866648 12460 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:52.867009 12460 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.858727 11349 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.859512 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:52.859638 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:52.850539 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:52.868649 11302 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.869098 11311 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:52.851285 12417 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:52.870182 12417 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:52.870568 12417 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.872345 12440 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:52.872978 12440 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:52.880070 12440 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:52.879719 12445 consensus_queue.cc:237] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.881798 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:52.882464 11383 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 2.
I20260321 02:58:52.883558 11942 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.888090 12440 raft_consensus.cc:2804] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:52.888528 12440 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:52.888860 12440 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.891031 12496 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.903201 12496 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:52.894758 11309 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:52.899194 12460 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.916453 12440 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:52.901052 12434 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.894002 12497 raft_consensus.cc:2804] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:58:52.908215 11302 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:52.921897 11976 raft_consensus.cc:1275] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260321 02:58:52.923581 11311 raft_consensus.cc:1275] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260321 02:58:52.929718 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:52.930348 11308 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.918701 12445 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:52.931329 12496 consensus_queue.cc:1048] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:58:52.930938 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:52.917037 12460 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:52.929068 12417 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.935449 12434 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:52.931017 12445 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.937992 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:52.938736 11230 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:52.939201 12477 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.941085 12477 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:52.941641 12477 consensus_queue.cc:1048] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:58:52.943308 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:52.944046 11309 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.944013 11302 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:52.945003 12497 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:52.945452 12497 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:52.946208 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:52.946779 11383 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:52.938208 12445 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:52.953286 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:52.953977 11302 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.958114 11349 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.959661 12497 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.962366 12497 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:52.957397 11383 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:58:52.960594 12445 raft_consensus.cc:2804] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:52.966537 12445 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:52.966987 12445 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.967075 11230 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:58:52.970278 12417 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:52.965668 11302 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:52.988873 12510 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:52.920037 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:52.989804 11236 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 2.
I20260321 02:58:52.974086 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:52.974736 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:52.975045 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:52.971897 11308 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:52.990557 11383 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.990675 11228 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.990854 11233 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:52.976979 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:52.977253 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:52.996731 11234 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:52.998483 11383 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:52.977509 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:52.977900 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:52.999692 11376 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.979246 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:52.999863 11200 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:52.999783 11232 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 3
I20260321 02:58:52.982694 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:52.983282 12440 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:52.985828 11309 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:52.986070 12445 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.989492 12510 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:52.976547 11275 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:52.991766 11306 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:58:53.020390 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 43937.4us
I20260321 02:58:53.021404 12510 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:53.022050 11234 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:58:53.022540 12507 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:52.996030 12422 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:53.023167 12507 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.023335 12422 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.023932 11232 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:53.001773 12417 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:53.025278 12422 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:53.026808 11306 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:58:53.008214 11305 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.008705 11378 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.001068 12418 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:53.031510 12418 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.025287 12507 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:53.012244 12511 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:53.035151 12511 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.037381 12511 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:53.010766 12445 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:53.022680 12433 raft_consensus.cc:2804] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:58:53.038790 12433 raft_consensus.cc:697] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 4 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:58:53.008245 11376 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:52.918864 12447 raft_consensus.cc:697] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:53.004292 11228 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:53.030861 11349 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:53.033571 12418 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:53.025043 12417 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.043761 12433 consensus_queue.cc:237] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.044713 12417 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:53.010383 12460 raft_consensus.cc:2804] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:53.046352 12460 raft_consensus.cc:697] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:53.047204 12460 consensus_queue.cc:237] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.048866 12447 consensus_queue.cc:237] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.063512 11349 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:53.064426 11349 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:53.064738 12445 raft_consensus.cc:2804] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:53.065953 12445 raft_consensus.cc:697] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:53.067338 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:53.041584 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:53.040221 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:53.068445 11308 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:53.068506 11311 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
W20260321 02:58:53.069494 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 64499.5us
I20260321 02:58:53.067617 12497 raft_consensus.cc:2804] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:53.070210 11200 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:53.070389 12497 raft_consensus.cc:697] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:53.071339 12497 consensus_queue.cc:237] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.071794 12422 raft_consensus.cc:2804] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:53.072827 12422 raft_consensus.cc:697] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:53.073781 11311 raft_consensus.cc:1275] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:58:53.067449 12445 consensus_queue.cc:237] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.074293 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:53.073738 12422 consensus_queue.cc:237] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.067869 12511 raft_consensus.cc:2804] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:58:53.076301 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:53.076601 12511 raft_consensus.cc:697] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:53.077558 12511 consensus_queue.cc:237] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.074911 11308 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:53.068866 11306 raft_consensus.cc:2468] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:53.068315 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:53.078540 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:53.128720 12418 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:53.120325 12496 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:53.124428 12440 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:53.120824 12510 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:53.129674 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:53.079792 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:53.123332 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:53.125218 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:53.126185 11307 raft_consensus.cc:1275] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260321 02:58:53.124086 12417 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:53.121968 12477 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:53.126868 12514 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:53.127381 11378 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:58:53.128902 11311 raft_consensus.cc:2393] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 2: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:53.130209 12440 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:53.129384 12418 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.125092 11305 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:53.130863 11228 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.130934 11303 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:53.121817 12507 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:53.130610 11383 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:53.131592 11309 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.132597 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:53.140554 11230 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:53.142397 12418 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:53.143025 11232 raft_consensus.cc:2468] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:53.145367 12507 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.148519 11228 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:53.137041 12417 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.133490 11349 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:53.133977 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:53.138305 11200 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:53.150573 11233 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:53.135010 12477 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.154734 12477 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:53.155272 12477 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:53.135773 12514 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.158591 12440 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:53.130216 12510 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.152326 12418 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:53.160724 12418 raft_consensus.cc:697] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:53.161803 12510 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:53.161875 12418 consensus_queue.cc:237] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.158705 12514 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:53.162595 12510 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:53.152395 12417 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:53.170068 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:53.170977 11233 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.173736 11309 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:53.175973 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:53.176836 11303 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:53.164382 11349 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:53.147715 12507 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:53.182670 12507 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:53.182565 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:53.183244 12515 consensus_queue.cc:1048] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:53.183938 11311 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:53.185118 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:58:53.191634 11233 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:58:53.176110 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:53.183511 12507 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.181129 11349 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:53.134166 12496 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.200055 11305 raft_consensus.cc:2468] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:53.176625 12510 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.168521 12477 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.221706 12515 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:53.224395 12515 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.226584 12515 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:53.220805 12496 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:53.223600 12440 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:53.245193 12496 raft_consensus.cc:2804] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:53.246399 12496 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:53.246852 12496 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.180166 11200 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:58:53.249248 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 69911us
I20260321 02:58:53.252279 12507 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:53.254014 12507 raft_consensus.cc:2804] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:53.254379 12507 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:53.254688 12507 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.252050 12440 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:53.253275 12510 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:53.256170 12515 raft_consensus.cc:2804] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:53.256532 12510 raft_consensus.cc:2804] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:53.256682 12515 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:53.257066 12510 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:53.257319 12515 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.257436 12510 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.257586 12440 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:53.261369 12477 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:53.261969 12477 consensus_queue.cc:1048] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:58:53.263052 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:53.263703 11311 raft_consensus.cc:2468] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 2.
I20260321 02:58:53.264855 11942 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:53.265619 12440 raft_consensus.cc:2804] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:53.265969 12440 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:53.266252 12440 raft_consensus.cc:3060] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.269629 11200 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:53.277072 11200 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:53.282778 12417 raft_consensus.cc:2804] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:53.283264 12417 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:53.283774 12417 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.282778 12514 raft_consensus.cc:2804] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:53.283943 12507 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.284904 12514 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:53.285326 12514 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.287781 12507 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:53.299180 12496 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.307175 12417 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.318269 12519 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:53.318903 12519 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.319921 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:53.288352 12440 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:53.295012 12510 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.322968 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:53.345896 11383 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:53.325392 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:53.327313 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:53.328837 11942 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:53.347196 11385 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:53.331509 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:53.290079 11200 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:53.348228 11376 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
W20260321 02:58:53.353294 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 158229us
I20260321 02:58:53.354422 11349 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:53.356235 11379 raft_consensus.cc:2393] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 3: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:58:53.345005 12507 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:53.357822 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:53.374991 12524 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:53.392906 12447 raft_consensus.cc:2804] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:53.367049 12507 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:58:53.427763 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 145121us
I20260321 02:58:53.430233 11200 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:53.399770 12487 raft_consensus.cc:2804] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:53.392899 11378 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 2.
I20260321 02:58:53.432744 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:53.433990 11233 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 2.
W20260321 02:58:53.435488 12516 log.cc:927] Time spent T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
W20260321 02:58:53.436859 12523 log.cc:927] Time spent T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
W20260321 02:58:53.438175 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 82781.4us
I20260321 02:58:53.406539 12417 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:53.440543 12417 raft_consensus.cc:2804] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:53.441009 12417 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:53.441411 12417 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.442193 12519 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:53.443918 12507 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:53.444504 12440 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:53.445313 12447 raft_consensus.cc:697] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:53.446229 12447 consensus_queue.cc:237] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.422606 12514 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.449011 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:53.449666 11311 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:53.450182 12514 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:53.450954 12514 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:53.420881 12524 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:53.452821 12496 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:53.453719 12487 raft_consensus.cc:697] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:58:53.454417 12524 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:53.454710 12487 consensus_queue.cc:237] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:53.458211 12510 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:53.459190 12515 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.463052 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:53.463758 11311 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.464166 12515 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:53.467005 12422 raft_consensus.cc:2804] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:53.467561 12422 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:53.451471 12514 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.469918 12514 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:53.468092 12422 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.480264 12514 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:53.480942 12514 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.483510 11311 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:53.484421 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:53.495510 11233 raft_consensus.cc:2393] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 3: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:53.486346 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:53.496214 11228 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
W20260321 02:58:53.430042 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 81873.1us
I20260321 02:58:53.492767 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:53.498216 11230 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:53.495146 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:53.499341 11236 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:53.483271 12514 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:53.523196 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:53.524123 11236 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.535782 11236 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:53.537461 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:53.538157 11232 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:53.539009 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:53.539263 12422 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.556738 11975 raft_consensus.cc:1240] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3. Ops: []
I20260321 02:58:53.560467 11942 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:58:53.506253 12445 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:53.601095 11234 raft_consensus.cc:1275] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:53.563131 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:53.592900 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:53.602077 12532 raft_consensus.cc:2804] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:53.626570 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:53.637589 11309 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:53.638674 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:53.639369 11309 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.627041 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:53.628043 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:53.640516 11311 raft_consensus.cc:3060] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.640722 11308 raft_consensus.cc:2393] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 2: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:53.628754 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:53.641229 11306 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:53.642133 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:53.642830 11306 raft_consensus.cc:2393] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 2: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:53.629091 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:53.643481 11304 raft_consensus.cc:2393] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 2: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:53.631265 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:53.644546 11310 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.642161 11233 raft_consensus.cc:1275] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:53.647500 11304 raft_consensus.cc:1275] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:53.649096 11304 raft_consensus.cc:1275] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:53.650962 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:53.651522 11308 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.654415 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:53.683208 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:53.685053 11303 raft_consensus.cc:2393] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 2: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:53.618232 12460 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:53.692793 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:53.693511 11233 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.693135 11309 raft_consensus.cc:2468] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:53.651310 11976 raft_consensus.cc:2368] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:58:53.637171 12532 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:53.703332 11307 raft_consensus.cc:2368] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:58:53.736327 12532 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.703377 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:53.737349 11379 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:53.704535 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:53.738575 11376 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.740401 12422 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:53.743101 12434 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:53.749642 11233 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:53.713224 11304 raft_consensus.cc:1275] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:53.646579 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:53.605243 12445 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.646579 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:58:53.718743 12433 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s user 0.012s sys 0.023s
I20260321 02:58:53.639987 12417 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.720710 12535 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:53.722870 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:53.712606 12460 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.754668 11311 raft_consensus.cc:2468] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
W20260321 02:58:53.719745 12530 log.cc:927] Time spent T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.124s user 0.001s sys 0.000s
I20260321 02:58:53.756779 11233 raft_consensus.cc:1275] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:53.759032 11376 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:53.767937 11309 raft_consensus.cc:1240] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260321 02:58:53.773120 11308 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
W20260321 02:58:53.756287 12433 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.580s user 0.013s sys 0.023s
I20260321 02:58:53.779731 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:53.780916 11305 raft_consensus.cc:2393] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 3: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:53.743809 12434 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.787822 11972 raft_consensus.cc:1275] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260321 02:58:53.774130 12535 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.797173 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:53.802733 11233 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:53.790870 11234 raft_consensus.cc:2368] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:58:53.809345 12535 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:53.817152 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:53.818022 11232 raft_consensus.cc:2393] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 3: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:53.818156 11310 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:53.797345 12433 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:53.806403 12524 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:58:53.858517 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 263079us
W20260321 02:58:53.865581 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 216633us
I20260321 02:58:53.870324 11232 raft_consensus.cc:1275] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:53.856994 11227 raft_consensus.cc:2368] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:58:53.875388 11200 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:53.883251 11200 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:53.879594 12542 consensus_queue.cc:1059] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260321 02:58:53.889119 12535 raft_consensus.cc:2804] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:53.896006 11350 leader_election.cc:400] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285) with higher term. Message: Invalid argument: T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:58:53.914929 11350 leader_election.cc:403] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260321 02:58:53.843894 12433 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.907599 11200 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:58:53.924094 11350 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 43859.9us
I20260321 02:58:53.864811 12524 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:53.927453 11349 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:53.931102 11942 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:53.939666 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:53.956036 11376 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.940658 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:53.959457 11379 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.943346 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:53.944802 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:53.961316 11385 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:53.953162 11381 raft_consensus.cc:1275] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:58:53.954946 11232 raft_consensus.cc:1275] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260321 02:58:53.917798 12514 raft_consensus.cc:2804] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:53.969672 11377 raft_consensus.cc:1275] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:53.908573 12535 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:53.970220 11349 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:53.926672 12510 raft_consensus.cc:3055] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:58:53.974705 11383 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:53.975447 11376 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:53.975524 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:53.978999 11377 raft_consensus.cc:2393] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 3: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:58:53.973567 11349 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:53.965144 11130 catalog_manager.cc:5671] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:53.980669 11349 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:53.981859 11349 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:53.986860 11377 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 3
I20260321 02:58:53.988898 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:53.965670 12445 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:54.018589 11376 raft_consensus.cc:2410] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 3 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260321 02:58:54.025367 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:53.965823 12541 raft_consensus.cc:3060] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 3
I20260321 02:58:54.023833 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:54.030598 11385 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:54.030671 11228 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:53.968053 12417 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:58:53.976950 12526 log.cc:927] Time spent T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.264s user 0.001s sys 0.000s
I20260321 02:58:53.971297 12514 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:53.976081 12535 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:53.989133 12546 raft_consensus.cc:2804] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:53.970898 12460 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:53.989897 11234 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2d71c397418a46d085c9974230b7f964"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:58:53.971076 12537 raft_consensus.cc:2804] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:53.990741 12447 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:53.991149 12496 raft_consensus.cc:2804] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:54.060422 11132 catalog_manager.cc:5671] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:54.064895 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:53.976317 12542 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:53.985108 12510 raft_consensus.cc:740] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:54.066699 12546 raft_consensus.cc:697] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:58:54.074985 11377 raft_consensus.cc:1275] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:54.040985 12442 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:54.031473 12434 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:54.034854 12521 raft_consensus.cc:2804] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:54.041253 12445 raft_consensus.cc:2804] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:54.117815 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:54.079268 11132 catalog_manager.cc:5671] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:58:54.118471 11302 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:54.045261 12534 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:54.058972 12514 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:54.123517 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:54.077657 12496 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:54.064385 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:54.098171 12537 raft_consensus.cc:697] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
W20260321 02:58:54.098873 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 117112us
I20260321 02:58:54.071167 11234 raft_consensus.cc:606] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:54.078164 12447 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:54.082580 12542 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:54.124373 11310 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:54.101380 12555 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:54.121625 11134 catalog_manager.cc:5671] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 1 to 3, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4). New cstate: current_term: 3 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:58:54.128679 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:54.139647 11309 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
W20260321 02:58:54.140640 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 117554us
I20260321 02:58:54.142057 11200 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:54.117776 12445 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:54.136694 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:58:54.137925 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 115373us
I20260321 02:58:54.124724 12434 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:54.133754 12496 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:54.136482 11302 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:54.151985 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:54.152480 11380 raft_consensus.cc:1275] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260321 02:58:54.151719 12547 consensus_queue.cc:1048] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:54.138988 12447 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:54.155157 11200 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:54.155380 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:54.159440 11309 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:54.152658 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:54.139858 11378 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 3
I20260321 02:58:54.120210 12442 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.141870 12542 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:54.171092 11303 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:54.176669 11200 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:54.150004 12445 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:54.185204 11200 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:54.130344 12540 consensus_queue.cc:1048] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260321 02:58:54.192691 11377 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b"
dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:44010
I20260321 02:58:54.193567 11377 raft_consensus.cc:606] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:54.191169 12460 raft_consensus.cc:2804] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:54.164093 12549 raft_consensus.cc:2804] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:54.210558 11379 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:54.191450 12417 raft_consensus.cc:2804] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:54.207348 11306 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:54.161732 12433 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:54.164378 12521 raft_consensus.cc:697] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:54.182297 12422 raft_consensus.cc:2804] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:54.129653 12534 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.223387 11309 raft_consensus.cc:1275] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:54.161896 12434 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:54.228750 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 95725.7us
I20260321 02:58:54.195349 12460 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:54.180608 12524 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:54.238116 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:54.238931 11310 raft_consensus.cc:2393] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 3: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:58:54.142907 12510 consensus_queue.cc:260] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:54.240844 11311 raft_consensus.cc:1275] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:54.146906 12555 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.216017 12417 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:54.241945 11379 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02"
dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
mode: GRACEFUL
new_leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
from {username='slave'} at 127.0.0.1:44010
I20260321 02:58:54.242677 11308 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:54.210861 12549 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:54.244812 11379 raft_consensus.cc:606] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Received request to transfer leadership to TS 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:58:54.248490 11379 raft_consensus.cc:614] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:58:54.185853 12548 consensus_queue.cc:1048] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260321 02:58:54.248920 11302 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:54.237185 12460 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:54.247753 12417 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:54.232893 12515 consensus_queue.cc:1048] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260321 02:58:54.252568 12549 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:54.280323 11380 raft_consensus.cc:1275] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:54.280740 11234 raft_consensus.cc:1275] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260321 02:58:54.290803 11380 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba"
dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
mode: GRACEFUL
new_leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
from {username='slave'} at 127.0.0.1:44010
I20260321 02:58:54.240118 12477 consensus_queue.cc:1048] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260321 02:58:54.291189 12555 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:54.291456 11380 raft_consensus.cc:606] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Received request to transfer leadership to TS 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:54.291921 11380 raft_consensus.cc:614] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:58:54.292309 12442 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:54.248950 12524 consensus_queue.cc:1048] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260321 02:58:54.294009 12434 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:54.294097 11303 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:54.266333 12510 raft_consensus.cc:3060] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:54.267438 12559 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:54.297056 12559 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.299279 11234 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:58:54.300472 11234 raft_consensus.cc:606] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:54.302114 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260321 02:58:54.303809 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:58:54.304641 11200 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:54.304606 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260321 02:58:54.305351 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 320038us
I20260321 02:58:54.306622 11200 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:54.308501 12422 raft_consensus.cc:697] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:54.309478 12422 consensus_queue.cc:237] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.177543 12546 consensus_queue.cc:237] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:54.241631 12537 consensus_queue.cc:237] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.239759 12556 consensus_queue.cc:1048] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
I20260321 02:58:54.321214 12568 raft_consensus.cc:2749] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:58:54.309509 11304 raft_consensus.cc:1275] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:54.329386 11306 raft_consensus.cc:1275] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260321 02:58:54.338625 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:54.338671 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:54.339263 11228 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
I20260321 02:58:54.342005 11303 raft_consensus.cc:1275] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:54.344648 11310 raft_consensus.cc:1275] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:54.348578 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:54.349742 11308 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:58:54.352887 11200 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:58:54.352950 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 100482us
I20260321 02:58:54.317193 12547 raft_consensus.cc:2804] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:54.359581 12534 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:54.299238 12559 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:54.356307 12568 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:54.376205 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:54.376634 11275 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:54.358888 12557 consensus_queue.cc:1048] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260321 02:58:54.367936 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:54.392206 12548 raft_consensus.cc:2749] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:58:54.393497 12547 raft_consensus.cc:697] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:54.400976 11200 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:54.274806 12521 consensus_queue.cc:237] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.404184 12564 consensus_queue.cc:1048] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260321 02:58:54.411681 12434 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260321 02:58:54.415041 12532 consensus_meta.cc:287] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 0.678s user 0.000s sys 0.012s
I20260321 02:58:54.415728 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:54.422822 12434 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:58:54.457302 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 223616us
I20260321 02:58:54.450919 12487 consensus_queue.cc:1048] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260321 02:58:54.458771 11349 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:54.432123 12557 raft_consensus.cc:2804] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:54.398566 12568 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:54.466336 12568 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:54.467247 12557 raft_consensus.cc:697] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:54.450488 12540 consensus_queue.cc:1048] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:54.467684 12541 raft_consensus.cc:2749] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285) with higher term. Message: Invalid argument: T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:58:54.476137 12507 raft_consensus.cc:2804] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:54.476660 12507 raft_consensus.cc:697] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:54.459620 12434 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:54.477627 12507 consensus_queue.cc:237] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.473774 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:54.474303 12496 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.476675 11381 raft_consensus.cc:1275] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260321 02:58:54.472674 12557 consensus_queue.cc:237] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.433225 12548 consensus_queue.cc:1048] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260321 02:58:54.499660 11349 leader_election.cc:400] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:58:54.503400 11349 leader_election.cc:403] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260321 02:58:54.504190 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:54.506371 11200 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:54.432435 12532 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:54.530146 11349 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:54.475692 12535 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.522611 12560 raft_consensus.cc:2749] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:58:54.524271 12496 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:58:54.512630 12570 log.cc:927] Time spent T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.097s user 0.003s sys 0.000s
I20260321 02:58:54.526458 12460 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.526715 12541 consensus_queue.cc:1048] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260321 02:58:54.512856 12568 raft_consensus.cc:2749] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:54.532285 11349 leader_election.cc:400] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:58:54.480504 12417 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.546551 11349 leader_election.cc:403] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260321 02:58:54.574527 11349 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:54.578336 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:54.578907 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:54.579010 11385 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
I20260321 02:58:54.579584 11382 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:54.583425 11381 raft_consensus.cc:1275] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260321 02:58:54.585611 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:54.586831 11382 raft_consensus.cc:2410] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 3 because replica has last-logged OpId of term: 1 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260321 02:58:54.588977 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:54.502703 12562 consensus_queue.cc:1048] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260321 02:58:54.536583 12477 raft_consensus.cc:2804] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:54.507459 12447 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.599040 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:54.488761 12542 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.601514 11230 raft_consensus.cc:2393] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 3: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:54.457551 12547 consensus_queue.cc:237] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.590983 12538 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:54.600941 12560 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:54.605518 12477 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:54.601318 11234 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
I20260321 02:58:54.664601 12477 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:54.665299 12565 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:54.665802 12565 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.671360 12572 consensus_queue.cc:1048] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.142s
I20260321 02:58:54.673924 11227 raft_consensus.cc:2393] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 3: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:54.680229 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:54.680337 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:54.681010 11303 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:54.681113 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:54.681348 11231 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:54.681707 11376 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:54.683079 11200 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:54.699265 12561 consensus_queue.cc:1048] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260321 02:58:54.659555 12560 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:54.650470 11378 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:54.680236 12565 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:54.710942 12563 raft_consensus.cc:2804] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:58:54.712329 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:54.675073 12532 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:54.707779 12496 raft_consensus.cc:2749] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:58:54.709817 12560 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:54.660811 12549 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.739528 12417 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:54.726292 12563 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:54.749408 12417 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:54.749604 11376 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:58:54.749943 12417 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.660221 12538 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.704540 12555 consensus_queue.cc:1048] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260321 02:58:54.755698 12542 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:54.758710 11233 raft_consensus.cc:2393] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 3: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:54.704253 12533 consensus_queue.cc:1048] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.147s
I20260321 02:58:54.754978 12460 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:54.752099 12535 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:54.765528 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:54.759684 12417 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:54.752245 11942 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:54.760747 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:54.754824 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:54.762591 11231 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:58:54.794906 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:58:54.752269 12514 consensus_meta.cc:287] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.616s user 0.011s sys 0.001s
I20260321 02:58:54.798314 12552 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:54.751379 12563 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:54.763721 12555 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:58:54.820330 12571 log.cc:927] Time spent T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.218s user 0.000s sys 0.001s
I20260321 02:58:54.745793 12568 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:58:54.833899 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 81719.2us
I20260321 02:58:54.838403 12546 raft_consensus.cc:2804] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:54.839104 12538 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:54.845041 12417 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:54.851907 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:58:54.809973 12445 consensus_meta.cc:287] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.592s user 0.012s sys 0.000s
I20260321 02:58:54.750571 12477 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.856536 12442 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:54.863322 12568 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.865620 12417 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.860314 11378 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 3
I20260321 02:58:54.853803 12514 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.846324 12447 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:54.875379 11376 raft_consensus.cc:1275] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260321 02:58:54.884346 11376 pending_rounds.cc:85] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5: Aborting all ops after (but not including) 4
I20260321 02:58:54.884713 11376 pending_rounds.cc:107] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260321 02:58:54.885012 11376 raft_consensus.cc:2889] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:58:54.866307 11231 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:58:54.888360 11275 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:54.889217 11382 raft_consensus.cc:2393] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 3: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:58:54.895397 12417 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:54.861626 12555 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.871452 12546 raft_consensus.cc:697] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:58:54.868904 12434 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:54.910861 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 191784us
I20260321 02:58:54.872282 12445 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.870612 12552 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.883522 12442 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.905330 12549 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:54.932401 11381 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
W20260321 02:58:54.936796 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 261861us
I20260321 02:58:54.937783 11349 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:58:54.918787 12407 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:54.939597 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:54.942445 11308 raft_consensus.cc:2393] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 2: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:54.942468 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:54.943399 11302 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:54.954892 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:54.954892 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:54.946084 11349 leader_election.cc:400] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267) with higher term. Message: Invalid argument: T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:58:54.940220 12518 raft_consensus.cc:2804] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:58:54.993906 11349 leader_election.cc:403] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260321 02:58:54.994679 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 48730.5us
I20260321 02:58:54.993211 11302 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:54.977885 11303 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:54.997130 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:58:55.004115 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 122835us
I20260321 02:58:54.955891 12524 raft_consensus.cc:2804] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:55.000489 11349 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:55.011276 11308 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:54.946961 12407 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:54.996318 12447 raft_consensus.cc:2749] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267) with higher term. Message: Invalid argument: T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:58:55.032763 12524 raft_consensus.cc:697] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:55.044238 12588 raft_consensus.cc:2749] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:58:55.044888 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:55.045783 11309 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:55.016539 12518 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:55.049574 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:55.050397 11307 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:55.053174 12524 consensus_queue.cc:237] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:55.051103 12541 consensus_queue.cc:1048] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260321 02:58:55.030321 12477 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:55.066612 11231 raft_consensus.cc:1275] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:54.983616 12546 consensus_queue.cc:237] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:55.060201 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:55.022171 12532 consensus_queue.cc:1048] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260321 02:58:55.046496 12515 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:55.070729 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:55.072062 11307 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:55.050202 12518 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:55.062045 11303 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:55.077941 11377 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:55.075587 11200 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:55.007522 12521 consensus_queue.cc:1048] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260321 02:58:55.050397 12514 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:55.081051 12417 raft_consensus.cc:2804] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:55.083194 12417 raft_consensus.cc:697] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:55.084247 12417 consensus_queue.cc:237] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:55.081806 11200 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:55.077890 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:55.076977 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:55.065894 12442 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:58:55.104168 12581 log.cc:927] Time spent T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.128s user 0.000s sys 0.001s
I20260321 02:58:55.071223 12447 consensus_queue.cc:1048] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260321 02:58:55.069777 12434 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:58:55.133764 12576 log.cc:927] Time spent T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20260321 02:58:55.075188 12568 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:58:55.138303 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 56567.7us
I20260321 02:58:55.142905 11309 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:55.066675 12510 consensus_queue.cc:1048] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260321 02:58:55.158236 11384 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:55.083920 12555 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:55.149775 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:55.189846 11377 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:55.161716 11305 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:55.107896 12445 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:55.203913 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:55.203943 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:55.158100 12514 raft_consensus.cc:2804] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:58:55.200889 11230 raft_consensus.cc:1275] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:55.084225 12515 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:55.159538 12541 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:55.209546 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:55.210464 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:55.211055 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:55.182480 12561 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:55.220440 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:55.216784 11230 raft_consensus.cc:2368] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:58:55.222702 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:55.223320 11377 raft_consensus.cc:1275] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:55.223667 11230 raft_consensus.cc:2393] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 3: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:55.216048 11231 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:55.218122 12550 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:55.217965 12541 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:55.236896 11236 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:55.215194 12514 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:55.248895 11235 raft_consensus.cc:1275] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260321 02:58:55.253249 12584 log.cc:927] Time spent T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260321 02:58:55.207988 12552 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:55.258304 11231 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
W20260321 02:58:55.263831 12582 log.cc:927] Time spent T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.225s user 0.001s sys 0.001s
W20260321 02:58:55.266180 12579 log.cc:927] Time spent T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.212s user 0.000s sys 0.001s
W20260321 02:58:55.268703 12586 log.cc:927] Time spent T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.084s user 0.000s sys 0.001s
I20260321 02:58:55.258235 12514 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:55.261178 12563 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:55.263028 12407 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:55.228960 12561 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:58:55.289566 12560 consensus_meta.cc:287] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.541s user 0.006s sys 0.008s
I20260321 02:58:55.290297 12541 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:55.290042 12560 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:58:55.292934 11236 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:55.257970 12550 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:55.304872 12525 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:55.307081 12550 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:55.306057 12525 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:55.309963 12525 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:55.331666 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:55.338106 11349 leader_election.cc:400] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 election: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:58:55.338515 11349 leader_election.cc:403] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260321 02:58:55.340843 11349 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:55.343218 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:55.337738 12510 consensus_queue.cc:1048] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:55.350399 12561 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:58:55.351118 12577 log.cc:927] Time spent T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.275s user 0.002s sys 0.000s
I20260321 02:58:55.346393 12541 consensus_queue.cc:1048] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260321 02:58:55.355427 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:55.325529 11305 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:55.380113 12541 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:55.381348 12541 raft_consensus.cc:697] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:55.390888 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:55.361130 12563 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:55.347325 12447 raft_consensus.cc:2749] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:58:55.409551 11200 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:58:55.410509 11231 raft_consensus.cc:1275] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:55.414520 11200 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:55.363191 11378 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
W20260321 02:58:55.417894 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 420915us
I20260321 02:58:55.386147 12560 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:55.354033 12445 consensus_queue.cc:1048] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20260321 02:58:55.396092 12533 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:55.438795 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:55.457736 11379 raft_consensus.cc:2393] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 3: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:58:55.436949 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:55.449072 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:55.454520 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:58:55.465592 12418 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.813s user 0.015s sys 0.079s
W20260321 02:58:55.466151 12418 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.840s user 0.015s sys 0.079s
I20260321 02:58:55.472694 11275 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:55.416249 12515 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:55.474840 11275 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:55.420616 12535 raft_consensus.cc:2804] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:55.417989 12460 raft_consensus.cc:2804] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:58:55.480700 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:55.481647 11231 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:55.382318 12541 consensus_queue.cc:237] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:55.478634 12552 raft_consensus.cc:2804] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:58:55.472354 12407 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:55.492761 12538 consensus_queue.cc:1048] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:55.480904 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:55.487491 12442 raft_consensus.cc:2764] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260321 02:58:55.487846 12552 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:55.486593 11234 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58214 (request call id 905) took 1147 ms. Trace:
I20260321 02:58:55.497834 12535 raft_consensus.cc:697] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:55.473973 12533 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:55.487823 12418 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:55.507267 12552 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
W20260321 02:58:55.544776 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 476278us
I20260321 02:58:55.549741 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:58:55.552218 11277 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.011s sys 0.023s
W20260321 02:58:55.552712 11277 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.527s user 0.011s sys 0.023s
I20260321 02:58:55.497969 12407 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:55.580123 11231 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:58:55.588891 12561 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:55.583356 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:55.564247 12418 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:55.611491 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:58:55.614934 11942 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.585s user 0.013s sys 0.022s
W20260321 02:58:55.615288 11942 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.586s user 0.013s sys 0.022s
I20260321 02:58:55.610105 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:58:55.615679 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 594329us
W20260321 02:58:55.622638 11302 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.020s sys 0.010s
W20260321 02:58:55.623109 11302 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.592s user 0.020s sys 0.010s
W20260321 02:58:55.614850 12574 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.772s user 0.013s sys 0.023s
W20260321 02:58:55.639613 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 213197us
I20260321 02:58:55.514830 11234 rpcz_store.cc:276] 0321 02:58:54.336638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:54.344024 (+ 7386us) service_pool.cc:225] Handling call
0321 02:58:55.477274 (+1133250us) spinlock_profiling.cc:243] Waited 203 ms on lock 0x7b5400263754. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:58:55.481790 (+ 4516us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":466872064}
I20260321 02:58:55.620384 11230 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
W20260321 02:58:55.650688 12574 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.808s user 0.014s sys 0.023s
I20260321 02:58:55.633304 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:55.698993 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:55.609741 12561 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:55.619796 12533 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:55.598995 12535 consensus_queue.cc:237] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:55.702448 11230 raft_consensus.cc:2468] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:58:55.706768 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:58:55.721571 11944 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.002s sys 0.037s
W20260321 02:58:55.721982 11944 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.702s user 0.003s sys 0.038s
W20260321 02:58:55.717149 12591 log.cc:927] Time spent T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.188s user 0.001s sys 0.000s
I20260321 02:58:55.724553 12407 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:55.690680 12548 consensus_queue.cc:1048] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260321 02:58:55.707795 11305 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:55.749701 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:55.750485 11302 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:55.751235 11234 raft_consensus.cc:2393] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 3: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:55.753628 11309 raft_consensus.cc:1275] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:55.758006 11349 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:55.761896 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:55.762584 11311 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:58:55.763377 12445 consensus_queue.cc:1048] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260321 02:58:55.763834 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:55.765496 12555 raft_consensus.cc:2804] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:55.765942 12555 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:55.766361 12555 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
W20260321 02:58:55.768083 11940 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.014s sys 0.028s
I20260321 02:58:55.768229 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:55.769131 11376 raft_consensus.cc:2393] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 3: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
W20260321 02:58:55.768628 11940 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.692s user 0.014s sys 0.028s
I20260321 02:58:55.784355 12418 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:55.779798 11200 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:55.789676 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:55.787667 11200 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:55.803541 11200 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:55.786409 12564 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:55.804970 12568 raft_consensus.cc:2804] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:55.811323 12568 raft_consensus.cc:697] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
W20260321 02:58:55.812860 12514 consensus_meta.cc:287] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.529s user 0.009s sys 0.003s
I20260321 02:58:55.812419 12568 consensus_queue.cc:237] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:55.818538 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 326042us
I20260321 02:58:55.821702 11275 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:55.803279 12548 raft_consensus.cc:2804] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:58:55.828552 11275 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:58:55.833818 12537 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.370s user 0.015s sys 0.029s
W20260321 02:58:55.834507 12537 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.371s user 0.015s sys 0.029s
W20260321 02:58:55.837806 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 96052.9us
I20260321 02:58:55.852401 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:55.852514 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:55.853193 11302 raft_consensus.cc:2368] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:58:55.854024 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:55.854569 11302 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:55.855031 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:55.855831 11311 raft_consensus.cc:1275] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:55.856011 11310 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:55.856218 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:55.809109 12515 raft_consensus.cc:2804] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:55.861829 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:55.869395 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:55.869671 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:55.870713 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:55.872846 11307 raft_consensus.cc:2368] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:58:55.813270 12514 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:55.849701 12407 raft_consensus.cc:2749] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:58:55.883320 12524 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:58:55.828339 12548 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:55.884637 12548 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:55.883983 12524 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:55.878525 12594 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:55.887115 12594 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:55.884074 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:55.817853 12564 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:55.933825 11235 raft_consensus.cc:1240] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260321 02:58:55.934296 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:58:55.965952 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 90935.3us
I20260321 02:58:55.972606 11311 raft_consensus.cc:2368] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:58:55.861505 12593 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:55.861177 12550 raft_consensus.cc:2749] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:58:55.970625 12515 raft_consensus.cc:697] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:55.973834 12445 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:55.974669 11303 raft_consensus.cc:2393] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 3: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:55.966778 11309 raft_consensus.cc:1275] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:55.849267 12460 raft_consensus.cc:697] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:55.982743 12445 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:55.983508 12460 consensus_queue.cc:237] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:55.984912 12445 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:55.942117 12434 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
W20260321 02:58:55.950454 12518 consensus_meta.cc:287] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.842s user 0.012s sys 0.003s
I20260321 02:58:55.987208 12434 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:55.987164 12518 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:55.979417 11310 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:55.881112 12595 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:55.990572 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:55.888840 12561 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:55.973006 11200 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:55.979189 11305 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:58:55.892956 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:55.966427 12524 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:58:56.451274 12555 consensus_meta.cc:287] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.684s user 0.006s sys 0.003s
I20260321 02:58:55.956677 12534 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:55.991406 11302 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:56.010911 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:56.541047 11307 raft_consensus.cc:1240] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260321 02:58:55.995424 11310 raft_consensus.cc:2393] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 3: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
W20260321 02:58:56.335217 11304 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.716s user 0.027s sys 0.043s
I20260321 02:58:56.539583 12534 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:56.078960 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:58:56.546823 11304 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.928s user 0.029s sys 0.047s
I20260321 02:58:55.928622 12594 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:56.565361 11229 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58200 (request call id 1310) took 1343 ms. Trace:
I20260321 02:58:56.591713 11229 rpcz_store.cc:276] 0321 02:58:55.198060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:55.198224 (+ 164us) service_pool.cc:225] Handling call
0321 02:58:55.218194 (+ 19970us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:58:55.218547 (+ 353us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:58:55.218577 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:58:55.219348 (+ 771us) log.cc:844] Serialized 26 byte log entry
0321 02:58:55.621827 (+402479us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:58:55.621867 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:55.621931 (+ 64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:58:56.541671 (+919740us) raft_consensus.cc:1685] finished
0321 02:58:56.541702 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:56.541858 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7667,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":10922,"thread_start_us":401954,"threads_started":1,"wal-append.queue_time_us":774664}
W20260321 02:58:56.565915 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 592975us
W20260321 02:58:56.529922 12548 consensus_meta.cc:287] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.645s user 0.009s sys 0.003s
I20260321 02:58:56.567380 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:56.552460 11311 raft_consensus.cc:2468] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:56.581363 11275 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:58:56.581935 11309 raft_consensus.cc:1275] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:56.077553 12550 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:55.941241 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:58:56.092929 12581 log.cc:927] Time spent T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
W20260321 02:58:56.463052 12552 consensus_meta.cc:287] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.893s user 0.010s sys 0.001s
I20260321 02:58:56.651741 12552 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:56.446290 12618 log.cc:927] Time spent T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.357s user 0.000s sys 0.001s
I20260321 02:58:56.652191 12612 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:56.010016 11227 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58214 (request call id 907) took 1672 ms. Trace:
I20260321 02:58:56.653098 12612 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:56.452942 12564 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:56.654096 11227 rpcz_store.cc:276] 0321 02:58:54.337263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:54.343306 (+ 6043us) service_pool.cc:225] Handling call
0321 02:58:54.415542 (+ 72236us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x7b5800111398. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b150e760 00007fa8b150e044 00007fa8aaa8f834 00007fa8aaa8f7c2
0321 02:58:56.009756 (+1594214us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b54003b9054. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:58:56.009948 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":457938560}
I20260321 02:58:56.608196 12538 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:56.655483 12538 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:56.656473 12612 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:56.658612 12538 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:56.233304 12515 consensus_queue.cc:237] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:56.650511 12601 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:56.664691 12601 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:56.600303 11228 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58200 (request call id 1318) took 1397 ms. Trace:
I20260321 02:58:56.668057 11228 rpcz_store.cc:276] 0321 02:58:55.202970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:55.266078 (+ 63108us) service_pool.cc:225] Handling call
0321 02:58:55.266529 (+ 451us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:58:55.266999 (+ 470us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:58:55.267040 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:58:55.268698 (+ 1658us) log.cc:844] Serialized 26 byte log entry
0321 02:58:55.691118 (+422420us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:58:55.691177 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:55.691275 (+ 98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:58:56.600078 (+908803us) raft_consensus.cc:1685] finished
0321 02:58:56.600100 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:56.600237 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":250090,"raft.queue_time_us":2301,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":3220,"thread_start_us":421975,"threads_started":1,"wal-append.queue_time_us":822475}
I20260321 02:58:56.651532 12550 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:56.674814 12550 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:56.671603 12601 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:56.654625 12564 raft_consensus.cc:2749] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:58:56.009909 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:56.361835 12546 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:56.257872 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:56.552281 12615 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:56.419394 12514 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:58:56.605764 11381 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40950 (request call id 913) took 2017 ms (client timeout 2089 ms). Trace:
I20260321 02:58:56.256021 12434 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:58:56.704046 11233 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58214 (request call id 911) took 2319 ms (client timeout 2089 ms). Trace:
I20260321 02:58:56.704579 12613 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:56.539382 12555 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:56.713979 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 118243us
I20260321 02:58:56.714962 11200 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:56.590212 12561 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
W20260321 02:58:56.711158 11381 rpcz_store.cc:269] 0321 02:58:54.589010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:54.598707 (+ 9697us) service_pool.cc:225] Handling call
0321 02:58:56.605483 (+2006776us) spinlock_profiling.cc:243] Waited 293 ms on lock 0x7b54001f68d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:58:56.605653 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":673798400}
I20260321 02:58:56.256283 12602 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:56.719525 12572 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:56.600152 12524 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:56.010331 12595 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:58:56.712447 11233 rpcz_store.cc:269] 0321 02:58:54.375484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:54.375660 (+ 176us) service_pool.cc:225] Handling call
0321 02:58:56.694216 (+2318556us) spinlock_profiling.cc:243] Waited 376 ms on lock 0x7b540027f1d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:58:56.694403 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":864807680}
I20260321 02:58:56.706911 12497 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:56.704598 12534 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:56.702755 12518 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:56.709327 12496 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:56.694077 12548 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:55.994791 12593 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:56.760630 11376 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.920s user 0.008s sys 0.049s
W20260321 02:58:56.749145 11231 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.963s user 0.013s sys 0.013s
W20260321 02:58:56.783084 11231 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.997s user 0.013s sys 0.013s
I20260321 02:58:56.782725 11200 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:56.784677 11200 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:58:56.782864 11376 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.943s user 0.009s sys 0.050s
I20260321 02:58:56.786890 11311 raft_consensus.cc:1275] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260321 02:58:56.743203 12547 raft_consensus.cc:2764] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260321 02:58:56.793677 11233 raft_consensus.cc:1275] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:56.797608 11233 raft_consensus.cc:1240] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260321 02:58:56.805377 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:56.709163 12546 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:56.810670 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:56.768836 12594 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:56.730144 12561 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:56.786284 12549 raft_consensus.cc:993] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb: : Instructing follower 83b12b883a7f4129a87dadf5a78f08da to start an election
I20260321 02:58:56.729616 12615 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:56.739018 12572 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:56.765619 12524 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:56.834692 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:56.771792 12496 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:56.754706 12497 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:56.737762 12613 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:56.787019 12564 consensus_queue.cc:1048] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260321 02:58:56.851015 12593 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:56.851626 12572 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:58:56.851727 12630 log.cc:927] Time spent T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.102s user 0.000s sys 0.001s
W20260321 02:58:56.853024 11277 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.639s user 0.000s sys 0.015s
I20260321 02:58:56.833299 12620 raft_consensus.cc:2804] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260321 02:58:56.866225 11277 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.656s user 0.000s sys 0.016s
I20260321 02:58:56.841917 12612 raft_consensus.cc:2804] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:58:56.823873 12594 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:56.876362 11380 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.048s user 0.025s sys 0.015s
I20260321 02:58:56.909704 11381 raft_consensus.cc:1275] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260321 02:58:56.910828 12642 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:56.911396 12572 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:56.912158 12572 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:56.911356 12642 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:56.914853 12572 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:56.914911 12642 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:56.865132 12567 raft_consensus.cc:1081] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Signalling peer 83b12b883a7f4129a87dadf5a78f08da to start an election
W20260321 02:58:56.924364 12599 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 1.011s user 0.005s sys 0.025s
I20260321 02:58:56.755192 12602 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:56.904546 11380 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.078s user 0.025s sys 0.015s
W20260321 02:58:56.933573 12599 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 1.037s user 0.005s sys 0.025s
I20260321 02:58:56.938889 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:56.898653 12552 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:56.870949 12620 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:56.917812 12612 raft_consensus.cc:697] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:56.955801 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:56.971262 11381 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:56.918030 12625 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
W20260321 02:58:56.974551 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 408196us
I20260321 02:58:56.975495 11275 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:58:56.931267 12537 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.001s sys 0.011s
I20260321 02:58:56.934207 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:56.941929 12562 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:58:56.973783 12568 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.003s user 0.015s sys 0.025s
W20260321 02:58:56.997538 12568 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.027s user 0.016s sys 0.025s
I20260321 02:58:56.998838 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:57.000166 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:57.000701 11311 raft_consensus.cc:2468] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
W20260321 02:58:56.934413 12521 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.161s user 0.000s sys 0.032s
W20260321 02:58:57.004174 12521 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.231s user 0.000s sys 0.033s
I20260321 02:58:57.007673 12445 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:57.008462 12445 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:57.012220 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:57.013087 11383 raft_consensus.cc:2468] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:57.014182 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:57.024559 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:56.932761 12555 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:58:57.023792 12537 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.763s user 0.001s sys 0.012s
W20260321 02:58:57.031287 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 55887us
I20260321 02:58:57.011116 12445 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:56.943128 12548 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:56.935770 12642 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:56.970685 12620 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:58:57.027706 12460 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.000s sys 0.015s
I20260321 02:58:56.939877 12639 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:57.037386 12534 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:58:57.039351 12595 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:57.033190 12593 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:58:57.046156 12460 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.607s user 0.000s sys 0.015s
I20260321 02:58:57.061496 12561 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:56.997581 12562 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:57.070026 12546 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:57.069322 12534 raft_consensus.cc:697] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:58:57.070966 12595 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:57.048660 12642 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:57.026466 12625 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:57.069963 12593 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:57.073444 12612 consensus_queue.cc:237] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:57.063383 12639 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:57.382966 11302 raft_consensus.cc:1240] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260321 02:58:57.515995 11379 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.603s user 0.019s sys 0.022s
W20260321 02:58:57.530582 11379 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s user 0.019s sys 0.031s
W20260321 02:58:57.374248 11303 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56616 (request call id 1352) took 1519 ms (client timeout 1786 ms). Trace:
I20260321 02:58:57.538758 12601 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:57.381702 11309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56616 (request call id 1360) took 1498 ms. Trace:
W20260321 02:58:57.329042 11201 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.191s user 0.005s sys 0.008s
I20260321 02:58:57.517530 11376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1186) took 1703 ms. Trace:
W20260321 02:58:57.328342 11199 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.090s user 0.006s sys 0.008s
W20260321 02:58:57.332581 11202 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.172s user 0.004s sys 0.024s
W20260321 02:58:57.516490 11942 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.694s user 0.017s sys 0.018s
W20260321 02:58:57.380890 11305 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.652s user 0.017s sys 0.008s
W20260321 02:58:57.356770 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 319501us
W20260321 02:58:57.554425 11201 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.417s user 0.016s sys 0.021s
I20260321 02:58:57.554823 12550 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:58:57.330044 11232 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.432s user 0.019s sys 0.012s
I20260321 02:58:57.555655 11275 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:58:57.556232 11232 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.658s user 0.026s sys 0.030s
I20260321 02:58:57.556048 12550 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:57.372501 11304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1180) took 1763 ms. Trace:
I20260321 02:58:57.329748 11231 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58200 (request call id 1327) took 1613 ms. Trace:
W20260321 02:58:57.329613 11229 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.000s sys 0.010s
W20260321 02:58:57.380890 11306 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.439s user 0.008s sys 0.020s
W20260321 02:58:57.538995 11303 rpcz_store.cc:269] 0321 02:58:55.854335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:55.857166 (+ 2831us) service_pool.cc:225] Handling call
0321 02:58:57.373645 (+1516479us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b5400319a54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:58:57.373976 (+ 331us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":239451904}
I20260321 02:58:57.349257 11235 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58200 (request call id 1315) took 2149 ms. Trace:
W20260321 02:58:57.524088 11385 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.003s sys 0.014s
I20260321 02:58:57.523252 11377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40950 (request call id 917) took 2433 ms. Trace:
W20260321 02:58:57.381949 11308 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56630 (request call id 1170) took 2385 ms (client timeout 2320 ms). Trace:
W20260321 02:58:57.566205 11306 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.624s user 0.012s sys 0.030s
W20260321 02:58:57.347883 11236 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58200 (request call id 1317) took 2145 ms (client timeout 1789 ms). Trace:
W20260321 02:58:57.384410 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 335146us
I20260321 02:58:57.383229 11307 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56616 (request call id 1350) took 1529 ms. Trace:
W20260321 02:58:57.566372 12590 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.102s user 0.013s sys 0.022s
W20260321 02:58:57.524379 11378 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.000s sys 0.018s
W20260321 02:58:57.569276 11199 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.331s user 0.014s sys 0.012s
I20260321 02:58:57.338301 11200 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:58:57.570138 11942 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.748s user 0.022s sys 0.021s
I20260321 02:58:57.568997 11377 rpcz_store.cc:276] 0321 02:58:55.089609 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:55.224752 (+135143us) service_pool.cc:225] Handling call
0321 02:58:55.225105 (+ 353us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:58:55.259385 (+ 34280us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:58:55.259434 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:58:55.259493 (+ 59us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:58:55.259537 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:57.522594 (+2263057us) spinlock_profiling.cc:243] Waited 34 ms on lock 0x7b5400099a54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:58:57.522689 (+ 95us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:57.523012 (+ 323us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":78223232}
W20260321 02:58:56.906699 11234 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.035s user 0.006s sys 0.023s
I20260321 02:58:57.567076 11235 rpcz_store.cc:276] 0321 02:58:55.199892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:55.260520 (+ 60628us) service_pool.cc:225] Handling call
0321 02:58:55.271103 (+ 10583us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:58:55.974466 (+703363us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:57.348483 (+1374017us) spinlock_profiling.cc:243] Waited 662 ms on lock 0x7b5400333bd4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:58:57.349022 (+ 539us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1523348864}
W20260321 02:58:57.622287 11305 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.893s user 0.022s sys 0.017s
I20260321 02:58:57.540290 12601 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:57.624262 12536 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:57.540485 11309 rpcz_store.cc:276] 0321 02:58:55.882631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:56.694756 (+812125us) service_pool.cc:225] Handling call
0321 02:58:56.716438 (+ 21682us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:58:56.716963 (+ 525us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:58:56.717004 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:58:56.717551 (+ 547us) log.cc:844] Serialized 26 byte log entry
0321 02:58:57.028838 (+311287us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:58:57.028878 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:57.028949 (+ 71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:58:57.380906 (+351957us) raft_consensus.cc:1685] finished
0321 02:58:57.380962 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:57.381377 (+ 415us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1401,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":17457,"thread_start_us":261937,"threads_started":1,"wal-append.queue_time_us":528364}
I20260321 02:58:57.538841 12655 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:58:57.154621 12629 log.cc:927] Time spent T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
W20260321 02:58:57.625685 11943 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.007s sys 0.005s
W20260321 02:58:57.215811 12658 log.cc:927] Time spent T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.061s user 0.002s sys 0.000s
W20260321 02:58:57.567533 11308 rpcz_store.cc:269] 0321 02:58:54.996746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:54.996882 (+ 136us) service_pool.cc:225] Handling call
0321 02:58:57.381400 (+2384518us) spinlock_profiling.cc:243] Waited 5.91 ms on lock 0x7b540018c254. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:58:57.381692 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13603584}
W20260321 02:58:57.568676 11236 rpcz_store.cc:269] 0321 02:58:55.202674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:55.207867 (+ 5193us) service_pool.cc:225] Handling call
0321 02:58:57.347230 (+2139363us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b5400061654. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:58:57.347544 (+ 314us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":47739264}
W20260321 02:58:57.627942 11229 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.875s user 0.007s sys 0.028s
W20260321 02:58:57.152256 12621 log.cc:927] Time spent T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.227s user 0.000s sys 0.001s
W20260321 02:58:57.622193 12407 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.782s user 0.002s sys 0.046s
W20260321 02:58:57.156860 12622 log.cc:927] Time spent T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.281s user 0.001s sys 0.001s
W20260321 02:58:57.240810 12628 log.cc:927] Time spent T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.147s user 0.000s sys 0.000s
I20260321 02:58:57.569555 11307 rpcz_store.cc:276] 0321 02:58:55.853843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:55.875990 (+ 22147us) service_pool.cc:225] Handling call
0321 02:58:56.540718 (+664728us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:58:56.541627 (+ 909us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:57.382645 (+841018us) spinlock_profiling.cc:243] Waited 661 ms on lock 0x7b540018ae50. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54bc8d 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:58:57.382992 (+ 347us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1521252608}
W20260321 02:58:57.253759 12633 log.cc:927] Time spent T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
W20260321 02:58:57.628681 11202 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.490s user 0.015s sys 0.034s
I20260321 02:58:57.636682 12644 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:57.637605 12624 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
W20260321 02:58:57.267262 12646 log.cc:927] Time spent T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.098s user 0.003s sys 0.000s
I20260321 02:58:57.566104 11231 rpcz_store.cc:276] 0321 02:58:55.741848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:55.753213 (+ 11365us) service_pool.cc:225] Handling call
0321 02:58:55.771532 (+ 18319us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:58:55.777603 (+ 6071us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:58:55.777648 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:58:55.786101 (+ 8453us) log.cc:844] Serialized 26 byte log entry
0321 02:58:56.783940 (+997839us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:58:56.784000 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:56.784072 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:58:57.328820 (+544748us) raft_consensus.cc:1685] finished
0321 02:58:57.328901 (+ 81us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:57.329317 (+ 416us) inbound_call.cc:173] Queueing success response
0321 02:58:57.528984 (+199667us) spinlock_profiling.cc:243] Waited 2.25 ms on lock 0x7b48000027d8. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b14721a1 00007fa8b1469237 00007fa8b14690ca 00007fa8b156092f 00007fa8b15605f7 00007fa8b1574c1c 00007fa8b1574bc8 00007fa8b1574b48 00007fa8b1574b03 00007fa8b1573dac 00007fa8b04acdf6
Metrics: {"mutex_wait_us":5620,"raft.queue_time_us":14124,"raft.run_cpu_time_us":36284,"raft.run_wall_time_us":1743290,"spinlock_wait_cycles":5186304,"thread_start_us":997432,"threads_started":1,"wal-append.queue_time_us":962634}
W20260321 02:58:57.260134 12607 log.cc:927] Time spent T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 1.453s user 0.013s sys 0.022s
W20260321 02:58:57.628000 11385 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s user 0.003s sys 0.020s
I20260321 02:58:57.083278 12524 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:57.558169 11304 rpcz_store.cc:276] 0321 02:58:55.608596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:55.608701 (+ 105us) service_pool.cc:225] Handling call
0321 02:58:55.608969 (+ 268us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:58:55.609166 (+ 197us) raft_consensus.cc:1534] Early marking committed up to index 2
0321 02:58:56.586988 (+977822us) log.cc:844] Serialized 14 byte log entry
0321 02:58:57.371555 (+784567us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:58:57.371671 (+ 116us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:58:57.371836 (+ 165us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:57.371985 (+ 149us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:57.372233 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":324515,"prepare.queue_time_us":1227006,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":54,"thread_start_us":1397928,"threads_started":2,"wal-append.queue_time_us":574323}
W20260321 02:58:57.695648 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 357744us
W20260321 02:58:57.646428 11227 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.795s user 0.013s sys 0.050s
W20260321 02:58:57.711454 11227 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.861s user 0.017s sys 0.066s
I20260321 02:58:57.646485 12518 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:57.712522 12518 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:57.714257 12644 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:57.715333 12521 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:57.716487 12521 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:57.717139 12644 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:57.111738 12615 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:57.622200 12555 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:58:57.632069 12590 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.247s user 0.013s sys 0.024s
W20260321 02:58:57.634699 11378 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.631s user 0.003s sys 0.027s
I20260321 02:58:57.720993 11305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56616 (request call id 1356) took 1865 ms. Trace:
W20260321 02:58:57.685091 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 835430us
I20260321 02:58:57.732900 12637 raft_consensus.cc:2804] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:58:57.278110 12602 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:58:57.708617 11943 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.815s user 0.008s sys 0.006s
I20260321 02:58:57.198985 12613 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:57.544577 11376 rpcz_store.cc:276] 0321 02:58:55.814198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:55.815827 (+ 1629us) service_pool.cc:225] Handling call
0321 02:58:55.819426 (+ 3599us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:58:55.819650 (+ 224us) raft_consensus.cc:1534] Early marking committed up to index 2
0321 02:58:56.788285 (+968635us) log.cc:844] Serialized 14 byte log entry
0321 02:58:57.061971 (+273686us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:58:57.516325 (+454354us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:58:57.516503 (+ 178us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:57.516693 (+ 190us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:57.517199 (+ 506us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":148313,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":24076,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":28349,"thread_start_us":1218635,"threads_started":2,"wal-append.queue_time_us":363699}
I20260321 02:58:57.720702 12555 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:57.215811 12594 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:57.080715 12496 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:58:57.731186 12407 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.966s user 0.003s sys 0.055s
I20260321 02:58:57.667382 12566 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:57.728451 11305 rpcz_store.cc:276] 0321 02:58:55.855571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:56.566537 (+710966us) service_pool.cc:225] Handling call
0321 02:58:56.581249 (+ 14712us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:58:56.583580 (+ 2331us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:58:56.583625 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:58:56.636594 (+ 52969us) log.cc:844] Serialized 26 byte log entry
0321 02:58:57.720301 (+1083707us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:58:57.720388 (+ 87us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:57.720671 (+ 283us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:58:57.720752 (+ 81us) raft_consensus.cc:1685] finished
0321 02:58:57.720765 (+ 13us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:57.720937 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2315,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":22126,"thread_start_us":894300,"threads_started":1,"wal-append.queue_time_us":554340}
I20260321 02:58:57.751149 11229 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58200 (request call id 1329) took 2008 ms. Trace:
I20260321 02:58:57.370131 11133 catalog_manager.cc:5671] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 2 to 4, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 4 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:58:57.744661 12637 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:58:57.768548 12648 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:57.769059 12637 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:57.769248 12552 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:57.769572 12648 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:57.768694 12562 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:57.770434 12521 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:57.741470 12644 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:57.780270 12422 raft_consensus.cc:2804] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:57.795722 12422 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:57.796373 12422 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:57.796464 12601 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:57.757968 12442 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:57.798920 12442 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:57.800957 12534 consensus_queue.cc:237] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:57.802361 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 247002us
I20260321 02:58:57.805168 12587 raft_consensus.cc:2749] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:58:57.775880 11200 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:57.797930 12596 raft_consensus.cc:493] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:57.801414 12445 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:57.270112 12620 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:57.740772 12624 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:57.848639 12620 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:57.768630 12597 raft_consensus.cc:493] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:58:57.817078 11234 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.794s user 0.012s sys 0.035s
I20260321 02:58:57.863229 11306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56616 (request call id 1348) took 2010 ms. Trace:
I20260321 02:58:57.762002 11229 rpcz_store.cc:276] 0321 02:58:55.742941 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:56.594384 (+851443us) service_pool.cc:225] Handling call
0321 02:58:56.594881 (+ 497us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:58:56.713507 (+118626us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:58:56.713551 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:58:56.720536 (+ 6985us) log.cc:844] Serialized 26 byte log entry
0321 02:58:57.667494 (+946958us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:58:57.667586 (+ 92us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:57.708370 (+ 40784us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:58:57.725442 (+ 17072us) raft_consensus.cc:1685] finished
0321 02:58:57.725459 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:57.751022 (+ 25563us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":121984,"raft.queue_time_us":134399,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":15360,"thread_start_us":895107,"threads_started":1,"wal-append.queue_time_us":423531}
I20260321 02:58:57.770364 12552 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:57.876549 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 138603us
I20260321 02:58:57.875895 11232 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59202 (request call id 66) took 3330 ms. Trace:
I20260321 02:58:57.884759 12639 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:57.890537 12642 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:57.775151 12536 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:58:57.904167 11233 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.915s user 0.029s sys 0.070s
I20260321 02:58:57.873214 11306 rpcz_store.cc:276] 0321 02:58:55.853231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:55.853815 (+ 584us) service_pool.cc:225] Handling call
0321 02:58:55.854257 (+ 442us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:58:55.862056 (+ 7799us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:58:55.862101 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:58:55.874728 (+ 12627us) log.cc:844] Serialized 26 byte log entry
0321 02:58:57.741948 (+1867220us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:58:57.764873 (+ 22925us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:57.774898 (+ 10025us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:58:57.788875 (+ 13977us) raft_consensus.cc:1685] finished
0321 02:58:57.788899 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:57.863146 (+ 74247us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7300,"raft.queue_time_us":21368,"raft.run_cpu_time_us":377,"raft.run_wall_time_us":62831,"thread_start_us":1626932,"threads_started":1,"wal-append.queue_time_us":1253765}
I20260321 02:58:57.925256 12556 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
W20260321 02:58:57.926178 11233 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.088s user 0.030s sys 0.070s
I20260321 02:58:57.893661 12644 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:57.935586 12649 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:57.891909 12620 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:57.932880 11310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56616 (request call id 1357) took 1909 ms. Trace:
I20260321 02:58:57.902386 12566 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:57.772887 12655 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:57.924868 12597 raft_consensus.cc:515] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:57.900439 12445 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:57.907945 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:57.888835 11232 rpcz_store.cc:276] 0321 02:58:54.541219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:54.541383 (+ 164us) service_pool.cc:225] Handling call
0321 02:58:54.603056 (+ 61673us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:58:54.620469 (+ 17413us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:58:54.620514 (+ 45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:58:54.638776 (+ 18262us) log.cc:844] Serialized 26 byte log entry
0321 02:58:55.897898 (+1259122us) spinlock_profiling.cc:243] Waited 18.4 ms on lock 0x7b18003b30e0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b2300441 00007fa8b230cca1 00007fa8a9498d17 00007fa8b2276b19 00007fa8a94b546b 00007fa8a94b370f 00007fa8a94b59d2 00007fa8ac0e55ab 00007fa8ac0eb0c3 00007fa8ac0eb471 00007fa8ac507def
0321 02:58:57.556892 (+1658994us) raft_consensus.cc:1645] Marking committed up to 4
0321 02:58:57.556952 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:57.557073 (+ 121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:58:57.810698 (+253625us) raft_consensus.cc:1685] finished
0321 02:58:57.810730 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:57.871724 (+ 60994us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6662,"raft.queue_time_us":7267,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":21061,"spinlock_wait_cycles":42259840,"thread_start_us":1658578,"threads_started":1,"wal-append.queue_time_us":2410606}
I20260321 02:58:57.973870 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:57.929565 12556 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:57.929639 12546 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:57.931383 12625 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:57.967865 12613 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:57.936486 12642 consensus_queue.cc:1048] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.159s
I20260321 02:58:58.091454 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:58.019420 11234 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58200 (request call id 1328) took 2276 ms. Trace:
I20260321 02:58:57.948276 12442 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.044315 12612 raft_consensus.cc:2804] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:58.101047 12655 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:58.101339 12611 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
W20260321 02:58:58.101838 12580 log.cc:927] Time spent T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 3.008s user 0.000s sys 0.026s
I20260321 02:58:58.038005 12601 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:58.038817 12538 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:58.110029 12496 consensus_queue.cc:1059] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:58:58.112423 12593 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.159009 12445 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:58.092900 12550 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:57.927753 12595 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:58.160204 11379 raft_consensus.cc:2468] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:58:57.934098 12596 raft_consensus.cc:515] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:58.107350 12612 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
W20260321 02:58:58.162554 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 59916.7us
I20260321 02:58:58.170374 12596 leader_election.cc:290] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:58:58.171167 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 164558us
I20260321 02:58:58.176868 12552 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.178021 11380 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45634 (request call id 63) took 2555 ms. Trace:
I20260321 02:58:58.180199 12538 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.181291 11378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45634 (request call id 65) took 2555 ms. Trace:
I20260321 02:58:58.182950 12538 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.181797 11378 rpcz_store.cc:276] 0321 02:58:55.625894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:55.639276 (+ 13382us) service_pool.cc:225] Handling call
0321 02:58:57.002493 (+1363217us) spinlock_profiling.cc:243] Waited 3.71 ms on lock 0x7b58002e3298. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b1522680 00007fa8b150cfe9 00007fa8b150ca92 00007fa8aaa8ad34
0321 02:58:57.002690 (+ 197us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:58:57.002922 (+ 232us) raft_consensus.cc:1534] Early marking committed up to index 5
0321 02:58:57.003148 (+ 226us) log.cc:844] Serialized 14 byte log entry
0321 02:58:58.088899 (+1085751us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:58:58.113732 (+ 24833us) raft_consensus.cc:1645] Marking committed up to 5
0321 02:58:58.180852 (+ 67120us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:58.180970 (+ 118us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:58.181221 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":7475,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":14566,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":267,"spinlock_wait_cycles":8536320,"thread_start_us":924344,"threads_started":2,"wal-append.queue_time_us":242241}
W20260321 02:58:58.184808 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 68910.9us
I20260321 02:58:58.185271 12496 raft_consensus.cc:3055] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:58:58.186028 12496 raft_consensus.cc:740] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:58:58.178514 11380 rpcz_store.cc:276] 0321 02:58:55.622684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:55.633220 (+ 10536us) service_pool.cc:225] Handling call
0321 02:58:55.701704 (+ 68484us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:58:55.745609 (+ 43905us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:58:55.762202 (+ 16593us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:58:55.806389 (+ 44187us) log.cc:844] Serialized 26 byte log entry
0321 02:58:56.983241 (+1176852us) raft_consensus.cc:1645] Marking committed up to 4
0321 02:58:56.995244 (+ 12003us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:57.523347 (+528103us) spinlock_profiling.cc:243] Waited 17.8 ms on lock 0x7b54001d8b54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:58:57.523440 (+ 93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:58:58.164069 (+640629us) raft_consensus.cc:1685] finished
0321 02:58:58.166348 (+ 2279us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:58.177849 (+ 11501us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1301,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":17345,"spinlock_wait_cycles":40992512,"thread_start_us":1111473,"threads_started":1,"wal-append.queue_time_us":1250068}
I20260321 02:58:58.072824 11310 rpcz_store.cc:276] 0321 02:58:55.863830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:56.582214 (+718384us) service_pool.cc:225] Handling call
0321 02:58:56.582596 (+ 382us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:58:56.807483 (+224887us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:58:56.807514 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:58:56.807990 (+ 476us) log.cc:844] Serialized 26 byte log entry
0321 02:58:57.055027 (+247037us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:58:57.055087 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:57.387221 (+332134us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b54003c91d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:58:57.387320 (+ 99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:58:57.722398 (+335078us) raft_consensus.cc:1685] finished
0321 02:58:57.722421 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:57.773353 (+ 50932us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1523,"raft.queue_time_us":6930,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":7840,"spinlock_wait_cycles":516318976,"thread_start_us":216092,"threads_started":1,"wal-append.queue_time_us":327044}
I20260321 02:58:58.154969 12536 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.187287 12496 consensus_queue.cc:260] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.208177 12496 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:58.208499 12595 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.156397 12655 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:58.189551 12612 raft_consensus.cc:3060] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:58.162678 12620 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.159857 12611 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.191874 12601 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:58.226909 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:58.073027 12649 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.215657 12555 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:58.246882 12611 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.259833 12587 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:58.255036 12644 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:58.243280 12536 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:58.271845 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:58.103235 11234 rpcz_store.cc:276] 0321 02:58:55.742647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:55.751921 (+ 9274us) service_pool.cc:225] Handling call
0321 02:58:55.752301 (+ 380us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:58:55.752555 (+ 254us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:58:55.752584 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:58:55.791634 (+ 39050us) log.cc:844] Serialized 26 byte log entry
0321 02:58:57.889620 (+2097986us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:58:57.889734 (+ 114us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:57.889860 (+ 126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:58:58.019125 (+129265us) raft_consensus.cc:1685] finished
0321 02:58:58.019150 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:58.019348 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":2012216,"threads_started":1,"wal-append.queue_time_us":1229778}
I20260321 02:58:58.224542 12546 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:58.261377 12636 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:58.260661 12587 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.251037 12655 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:58.289086 11302 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56616 (request call id 1359) took 2409 ms. Trace:
I20260321 02:58:58.289934 11302 rpcz_store.cc:276] 0321 02:58:55.880108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:56.598479 (+718371us) service_pool.cc:225] Handling call
0321 02:58:57.382126 (+783647us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:58:57.529631 (+147505us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:58.284887 (+755256us) spinlock_profiling.cc:243] Waited 783 ms on lock 0x7b540018c250. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54bc8d 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:58:58.289014 (+ 4127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1800899456}
W20260321 02:58:58.284759 11349 connection.cc:232] RPC call timeout handler was delayed by -1.2190092919999529s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2097s for a response
W20260321 02:58:58.294814 11349 connection.cc:232] RPC call timeout handler was delayed by -1.2145092919999456s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2102s for a response
W20260321 02:58:58.295264 11349 connection.cc:232] RPC call timeout handler was delayed by -1.1442371509999703s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1835s for a response
W20260321 02:58:58.295562 11349 connection.cc:232] RPC call timeout handler was delayed by -1.1325558209999258s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2155s for a response
W20260321 02:58:58.295814 11349 connection.cc:232] RPC call timeout handler was delayed by -1.10560929199994s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2223s for a response
I20260321 02:58:58.257936 12613 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:58.298239 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:58.293831 12565 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:58.299306 12597 leader_election.cc:290] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:58:58.299849 11349 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.619s (SENT) [suppressed 1 similar messages]
W20260321 02:58:58.298265 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 80162us
I20260321 02:58:58.309108 12536 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.309746 12418 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:58.324601 12566 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:58:58.325735 12669 log.cc:927] Time spent T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.400s user 0.003s sys 0.000s
I20260321 02:58:58.337854 11229 raft_consensus.cc:2393] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 4: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
W20260321 02:58:58.329043 12422 consensus_meta.cc:287] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.531s user 0.008s sys 0.006s
I20260321 02:58:58.344944 11979 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:58.346246 12565 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.341840 12422 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.348953 12418 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:58.350323 12556 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.351342 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:58.357023 12546 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:58.327080 11973 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:58.357674 12566 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:58.358253 12597 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:58:58.325510 12636 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.340924 12601 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:58:58.362167 12637 consensus_meta.cc:287] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.592s user 0.000s sys 0.014s
I20260321 02:58:58.371695 12566 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.372728 12422 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.374140 12566 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.338819 12639 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:58:58.381810 12587 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.385105 12611 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:58.385198 12640 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:58.386170 12611 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:58.388844 12536 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.390376 12655 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:58.390484 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:58.391520 11973 raft_consensus.cc:2468] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
W20260321 02:58:58.392867 12657 log.cc:927] Time spent T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.501s user 0.000s sys 0.004s
I20260321 02:58:58.393060 12613 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:58.395831 11276 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:58:58.391526 12672 log.cc:927] Time spent T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.207s user 0.001s sys 0.001s
I20260321 02:58:58.391537 12620 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.407717 12679 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:58:58.408088 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:58.408596 12679 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:58.409829 11233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58200 (request call id 1332) took 1610 ms. Trace:
I20260321 02:58:58.374989 12597 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:58.404021 11201 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:58:58.395871 12611 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:58.416777 12644 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
W20260321 02:58:58.417691 12680 log.cc:927] Time spent T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20260321 02:58:58.386523 12640 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.412312 12536 raft_consensus.cc:2804] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:58.411792 11233 rpcz_store.cc:276] 0321 02:58:56.799376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:56.801058 (+ 1682us) service_pool.cc:225] Handling call
0321 02:58:56.801510 (+ 452us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:58:56.801954 (+ 444us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:58:56.801994 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:58:56.808266 (+ 6272us) log.cc:844] Serialized 26 byte log entry
0321 02:58:58.104408 (+1296142us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:58:58.122643 (+ 18235us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:58.164015 (+ 41372us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:58:58.391834 (+227819us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b440002304c. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93c2dc5 00007fa8a93c37e2 00007fa8ac0f757b 00007fa8ac0f2e4f 00007fa8ac0e6c0e 00007fa8ac0e5f56 00007fa8ac0e5b78 00007fa8ac0ef5c1 00007fa8ac0ef579 00007fa8ac0ef509
0321 02:58:58.395549 (+ 3715us) raft_consensus.cc:1685] finished
0321 02:58:58.395570 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:58.409754 (+ 14184us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":46059,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":23811,"spinlock_wait_cycles":498643200,"thread_start_us":1094594,"threads_started":1,"wal-append.queue_time_us":1065407}
I20260321 02:58:58.377681 12637 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.440579 12640 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.418197 12518 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.405745 12422 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:58.421128 12524 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:58:58.411913 12639 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:58.454674 12639 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:58.456148 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:58.357941 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:58:58.458194 11134 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44688 (request call id 124) took 4390 ms. Trace:
I20260321 02:58:58.461547 12535 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:58.418522 12655 consensus_queue.cc:1048] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260321 02:58:58.427549 12536 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:58.435292 12611 raft_consensus.cc:2804] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:58.427335 12644 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:58:58.481571 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 181764us
I20260321 02:58:58.466756 12536 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:58.465606 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:58.448516 12643 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:58.464423 12649 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.491456 12643 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.494167 12643 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:58.468858 12611 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:58.487476 11942 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:58.476070 12460 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:58.465178 12535 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.496881 11973 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:58.505163 12565 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.505357 12612 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.514838 12679 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:58.458875 11134 rpcz_store.cc:276] 0321 02:58:54.067255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:54.067410 (+ 155us) service_pool.cc:225] Handling call
0321 02:58:58.457871 (+4390461us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:58:54.671357 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:58:54.671930 (+ 573us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:58:54.672788 (+ 858us) write_op.cc:435] Acquired schema lock
0321 02:58:54.672809 (+ 21us) tablet.cc:661] Decoding operations
0321 02:58:54.673161 (+ 352us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:58:54.673212 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0321 02:58:54.673242 (+ 30us) tablet.cc:684] Acquiring locks for 1 operations
0321 02:58:54.673438 (+ 196us) tablet.cc:700] Row locks acquired
0321 02:58:54.673459 (+ 21us) write_op.cc:260] PREPARE: finished
0321 02:58:54.673595 (+ 136us) write_op.cc:270] Start()
0321 02:58:54.673702 (+ 107us) write_op.cc:276] Timestamp: P: 1774061934673547 usec, L: 0
0321 02:58:54.673735 (+ 33us) op_driver.cc:348] REPLICATION: starting
0321 02:58:54.674338 (+ 603us) log.cc:844] Serialized 530 byte log entry
0321 02:58:57.236311 (+2561973us) op_driver.cc:464] REPLICATION: finished
0321 02:58:57.256665 (+ 20354us) write_op.cc:301] APPLY: starting
0321 02:58:57.256754 (+ 89us) tablet.cc:1366] starting BulkCheckPresence
0321 02:58:57.256887 (+ 133us) tablet.cc:1369] finished BulkCheckPresence
0321 02:58:57.256998 (+ 111us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:58:57.257235 (+ 237us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:58:57.257430 (+ 195us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0321 02:58:57.257517 (+ 87us) write_op.cc:312] APPLY: finished
0321 02:58:57.257750 (+ 233us) log.cc:844] Serialized 23 byte log entry
0321 02:58:57.257961 (+ 211us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:58:57.258063 (+ 102us) write_op.cc:454] Released schema lock
0321 02:58:57.258129 (+ 66us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":41097,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":20066,"apply.run_cpu_time_us":1802,"apply.run_wall_time_us":1801,"num_ops":1,"prepare.queue_time_us":529386,"prepare.run_cpu_time_us":2431,"prepare.run_wall_time_us":3221,"replication_time_us":2562412,"thread_start_us":269826,"threads_started":1}]]}
I20260321 02:58:58.488231 12422 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.487232 12648 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:58.516283 11232 raft_consensus.cc:1240] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:58.517061 11201 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
W20260321 02:58:58.545735 11349 connection.cc:232] RPC call timeout handler was delayed by -1.0274263889999702s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19440000000000002s for a response
W20260321 02:58:58.554522 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 74397.3us
W20260321 02:58:58.620158 12684 log.cc:927] Time spent T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
I20260321 02:58:58.692543 12597 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:58.618058 11385 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1191) took 1620 ms. Trace:
W20260321 02:58:58.641336 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 124395us
I20260321 02:58:58.499532 12611 raft_consensus.cc:3060] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
W20260321 02:58:58.546577 12670 log.cc:927] Time spent T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.622s user 0.002s sys 0.000s
W20260321 02:58:58.671584 12683 log.cc:927] Time spent T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.206s user 0.001s sys 0.000s
I20260321 02:58:58.517590 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:58.719601 11311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1194) took 1801 ms. Trace:
W20260321 02:58:58.719937 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:58:58.720093 11311 rpcz_store.cc:276] 0321 02:58:56.865534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:57.385693 (+520159us) service_pool.cc:225] Handling call
0321 02:58:57.644246 (+258553us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:58:57.803784 (+159538us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:58:57.803818 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:58:57.804431 (+ 613us) log.cc:844] Serialized 26 byte log entry
0321 02:58:57.880495 (+ 76064us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:58:57.880547 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:58.390611 (+510064us) spinlock_profiling.cc:243] Waited 8.31 ms on lock 0x7b440002304c. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93c2dc5 00007fa8a93c37e2 00007fa8ac0f757b 00007fa8ac0f2e4f 00007fa8ac0e6c0e 00007fa8ac0e5f56 00007fa8ac0e5b78 00007fa8ac0ef5c1 00007fa8ac0ef579 00007fa8ac0ef509
0321 02:58:58.632957 (+242346us) spinlock_profiling.cc:243] Waited 91.2 ms on lock 0x7b54003c82d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:58:58.633016 (+ 59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:58:58.633063 (+ 47us) raft_consensus.cc:1685] finished
0321 02:58:58.633076 (+ 13us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:58.666517 (+ 33441us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":228850176,"thread_start_us":75551,"threads_started":1,"wal-append.queue_time_us":290641}
I20260321 02:58:58.508483 12524 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:58.549453 12644 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:58.723254 12524 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:58.547035 12636 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.667819 12679 raft_consensus.cc:2804] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:58.516033 12612 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.672302 12637 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.595714 12497 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:58:58.718081 12673 log.cc:927] Time spent T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.526s user 0.003s sys 0.000s
I20260321 02:58:58.713661 12445 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:58.594668 12418 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:58.591518 12572 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:58.734521 11303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1195) took 1834 ms. Trace:
I20260321 02:58:58.693310 11385 rpcz_store.cc:276] 0321 02:58:56.997871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:56.998096 (+ 225us) service_pool.cc:225] Handling call
0321 02:58:56.998394 (+ 298us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:58:57.009789 (+ 11395us) raft_consensus.cc:1534] Early marking committed up to index 2
0321 02:58:57.010104 (+ 315us) log.cc:844] Serialized 14 byte log entry
0321 02:58:58.023157 (+1013053us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:58:58.079868 (+ 56711us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:58:58.102156 (+ 22288us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:58.617814 (+515658us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b54000ce254. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:58:58.617861 (+ 47us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:58.617998 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":59146,"prepare.queue_time_us":808,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":64449,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"spinlock_wait_cycles":25673472,"thread_start_us":783766,"threads_started":2,"wal-append.queue_time_us":259694,"wal-append.run_cpu_time_us":1613,"wal-append.run_wall_time_us":1028386}
I20260321 02:58:58.741328 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:58.743112 12433 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:58.742762 12445 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.743855 12433 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.745636 12445 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:58.746066 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:58.746279 12433 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.738176 11303 rpcz_store.cc:276] 0321 02:58:56.900204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:57.570006 (+669802us) service_pool.cc:225] Handling call
0321 02:58:58.346102 (+776096us) spinlock_profiling.cc:243] Waited 877 us on lock 0x7b5800002a98. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b1522680 00007fa8b150cfe9 00007fa8b150ca92 00007fa8aaa8ad34
0321 02:58:58.360793 (+ 14691us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:58:58.375842 (+ 15049us) raft_consensus.cc:1534] Early marking committed up to index 2
0321 02:58:58.501611 (+125769us) log.cc:844] Serialized 14 byte log entry
0321 02:58:58.734149 (+232538us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:58:58.734225 (+ 76us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:58:58.734266 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:58.734332 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:58.734465 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":253162,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":7475,"spinlock_wait_cycles":2016512,"thread_start_us":191924,"threads_started":2}
I20260321 02:58:58.751936 11230 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58200 (request call id 1326) took 3052 ms. Trace:
I20260321 02:58:58.752462 11230 rpcz_store.cc:276] 0321 02:58:55.699305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:55.718155 (+ 18850us) service_pool.cc:225] Handling call
0321 02:58:55.718541 (+ 386us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:58:55.718900 (+ 359us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:58:55.718940 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:58:55.721620 (+ 2680us) log.cc:844] Serialized 26 byte log entry
0321 02:58:56.165785 (+444165us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:58:56.165848 (+ 63us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:56.165977 (+ 129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:58:58.750006 (+2584029us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x7b2400101198. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93088f4 00007fa8a93069fb 00007fa8a9395799 00007fa8a9395425 00007fa8a9394980 00007fa8ac113828 00007fa8ac113f88 00007fa8ac0eb738 00007fa8ac0e6b0a 00007fa8ac0e5f56
0321 02:58:58.751333 (+ 1327us) raft_consensus.cc:1685] finished
0321 02:58:58.751356 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:58.751756 (+ 400us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":57510,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":11833,"spinlock_wait_cycles":35779456,"thread_start_us":443679,"threads_started":1,"wal-append.queue_time_us":1111892}
I20260321 02:58:58.754757 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:58.755159 12535 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.755496 11230 raft_consensus.cc:2468] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
I20260321 02:58:58.729665 12679 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:58.517055 12460 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.736476 12422 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.760814 12679 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:58.768016 12572 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.777315 11381 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1190) took 1804 ms. Trace:
I20260321 02:58:58.777742 12572 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:58.778543 12611 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:58.777808 11381 rpcz_store.cc:276] 0321 02:58:56.973194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:56.982107 (+ 8913us) service_pool.cc:225] Handling call
0321 02:58:56.982580 (+ 473us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:58:57.768881 (+786301us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:58:57.768925 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:58:57.769611 (+ 686us) log.cc:844] Serialized 26 byte log entry
0321 02:58:58.045039 (+275428us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:58:58.045096 (+ 57us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:58.776761 (+731665us) spinlock_profiling.cc:243] Waited 756 ms on lock 0x7b54001f63d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:58:58.776801 (+ 40us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:58:58.776869 (+ 68us) raft_consensus.cc:1685] finished
0321 02:58:58.776883 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:58:58.777149 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1739293056,"thread_start_us":200347,"threads_started":1,"wal-append.queue_time_us":306810}
I20260321 02:58:58.781478 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:58.782294 11378 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 1.
I20260321 02:58:58.786162 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:58.787410 11130 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42502 (request call id 221) took 5166 ms. Trace:
I20260321 02:58:58.790815 11132 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42530 (request call id 213) took 5179 ms. Trace:
W20260321 02:58:58.769498 11975 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.006s sys 0.032s
I20260321 02:58:58.790591 11942 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:58.796857 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:58:58.792147 11975 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.606s user 0.006s sys 0.033s
I20260321 02:58:58.769013 12497 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:58.802584 12497 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.800210 11230 raft_consensus.cc:1275] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260321 02:58:58.804692 12685 log.cc:927] Time spent T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.178s user 0.004s sys 0.004s
I20260321 02:58:58.787808 11130 rpcz_store.cc:276] 0321 02:58:53.654140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:53.703782 (+ 49642us) service_pool.cc:225] Handling call
0321 02:58:58.787223 (+5083441us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:58:54.233042 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:58:54.323124 (+ 90082us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:58:54.323165 (+ 41us) write_op.cc:435] Acquired schema lock
0321 02:58:54.323186 (+ 21us) tablet.cc:661] Decoding operations
0321 02:58:54.323592 (+ 406us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:58:54.323673 (+ 81us) write_op.cc:641] Partition lock acquired for write op
0321 02:58:54.323708 (+ 35us) tablet.cc:684] Acquiring locks for 1 operations
0321 02:58:54.323893 (+ 185us) tablet.cc:700] Row locks acquired
0321 02:58:54.323917 (+ 24us) write_op.cc:260] PREPARE: finished
0321 02:58:54.344067 (+ 20150us) write_op.cc:270] Start()
0321 02:58:54.356539 (+ 12472us) write_op.cc:276] Timestamp: P: 1774061934344004 usec, L: 0
0321 02:58:54.362652 (+ 6113us) op_driver.cc:348] REPLICATION: starting
0321 02:58:54.415618 (+ 52966us) log.cc:844] Serialized 535 byte log entry
0321 02:58:55.895932 (+1480314us) op_driver.cc:464] REPLICATION: finished
0321 02:58:57.069145 (+1173213us) write_op.cc:301] APPLY: starting
0321 02:58:57.077133 (+ 7988us) tablet.cc:1366] starting BulkCheckPresence
0321 02:58:57.083901 (+ 6768us) tablet.cc:1369] finished BulkCheckPresence
0321 02:58:57.121480 (+ 37579us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:58:57.151296 (+ 29816us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:58:57.204699 (+ 53403us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0321 02:58:57.231259 (+ 26560us) write_op.cc:312] APPLY: finished
0321 02:58:57.232290 (+ 1031us) log.cc:844] Serialized 23 byte log entry
0321 02:58:57.517248 (+284958us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:58:57.517335 (+ 87us) write_op.cc:454] Released schema lock
0321 02:58:58.454741 (+937406us) spinlock_profiling.cc:243] Waited 27.9 ms on lock 0x7b1800303f40. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b2300441 00007fa8b230cca1 00007fa8ac0f385c 00007fa8a949939b 00007fa8b2276b19 00007fa8a94b546b 00007fa8a94b370f 00007fa8a94b59d2 00007fa8ac0e55ab 00007fa8ac0eb0c3 00007fa8ac0eb5da
0321 02:58:58.455046 (+ 305us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:58:58.455172 (+ 126us) write_op.cc:454] Released schema lock
0321 02:58:58.455243 (+ 71us) write_op.cc:341] FINISH: Updating metrics
0321 02:58:58.455808 (+ 565us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:58:58.455881 (+ 73us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6229,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":1152909,"apply.run_cpu_time_us":27642,"apply.run_wall_time_us":1392981,"num_ops":1,"prepare.queue_time_us":203697,"prepare.run_cpu_time_us":18877,"prepare.run_wall_time_us":382428,"raft.queue_time_us":3833700,"raft.run_cpu_time_us":62236,"raft.run_wall_time_us":1579297,"replication_time_us":1533108,"spinlock_wait_cycles":64172416,"thread_start_us":2526896,"threads_started":5,"wal-append.queue_time_us":404760,"wal-append.run_cpu_time_us":46722,"wal-append.run_wall_time_us":3029242}]]}
I20260321 02:58:58.806525 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:58.807232 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:58.810267 12497 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:58.810694 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:58.812510 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:58.812935 12624 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:58.814498 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:58.817051 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:58.818243 11234 raft_consensus.cc:2393] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:58.818681 12440 raft_consensus.cc:2804] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:58.819141 12440 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:58.819564 12440 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:58.819787 12497 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:58.820719 12497 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.822985 12497 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:58:58.817173 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:58:58.829131 12689 log.cc:927] Time spent T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.203s user 0.013s sys 0.000s
I20260321 02:58:58.830112 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:58.831151 11230 raft_consensus.cc:2410] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260321 02:58:58.791167 11132 rpcz_store.cc:276] 0321 02:58:53.611213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:53.611457 (+ 244us) service_pool.cc:225] Handling call
0321 02:58:58.790035 (+5178578us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:58:54.661601 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:58:54.665120 (+ 3519us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:58:54.665188 (+ 68us) write_op.cc:435] Acquired schema lock
0321 02:58:54.665238 (+ 50us) tablet.cc:661] Decoding operations
0321 02:58:54.665807 (+ 569us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:58:54.665871 (+ 64us) write_op.cc:641] Partition lock acquired for write op
0321 02:58:54.665913 (+ 42us) tablet.cc:684] Acquiring locks for 2 operations
0321 02:58:54.666302 (+ 389us) tablet.cc:700] Row locks acquired
0321 02:58:54.666329 (+ 27us) write_op.cc:260] PREPARE: finished
0321 02:58:54.666502 (+ 173us) write_op.cc:270] Start()
0321 02:58:54.666611 (+ 109us) write_op.cc:276] Timestamp: P: 1774061934666450 usec, L: 0
0321 02:58:54.666630 (+ 19us) op_driver.cc:348] REPLICATION: starting
0321 02:58:54.667406 (+ 776us) log.cc:844] Serialized 917 byte log entry
0321 02:58:57.061300 (+2393894us) op_driver.cc:464] REPLICATION: finished
0321 02:58:57.241415 (+180115us) write_op.cc:301] APPLY: starting
0321 02:58:57.241543 (+ 128us) tablet.cc:1366] starting BulkCheckPresence
0321 02:58:57.252562 (+ 11019us) tablet.cc:1369] finished BulkCheckPresence
0321 02:58:57.254070 (+ 1508us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:58:57.254683 (+ 613us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:58:57.254927 (+ 244us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0321 02:58:57.255002 (+ 75us) write_op.cc:312] APPLY: finished
0321 02:58:57.255276 (+ 274us) log.cc:844] Serialized 29 byte log entry
0321 02:58:57.255480 (+ 204us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:58:57.255605 (+ 125us) write_op.cc:454] Released schema lock
0321 02:58:57.255659 (+ 54us) write_op.cc:341] FINISH: Updating metrics
0321 02:58:57.256152 (+ 493us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:58:57.256196 (+ 44us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1246661,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":173550,"apply.run_cpu_time_us":2672,"apply.run_wall_time_us":14727,"num_ops":2,"prepare.queue_time_us":500496,"prepare.run_cpu_time_us":3364,"prepare.run_wall_time_us":12009,"replication_time_us":2381421,"thread_start_us":82338,"threads_started":1}]]}
I20260321 02:58:58.829239 12611 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:58.844084 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:58.844306 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:58.860216 11381 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 1.
I20260321 02:58:58.862018 11942 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:58.792191 12536 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:58.868250 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:58.843525 12497 consensus_queue.cc:1048] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260321 02:58:58.863525 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:58.882889 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:58:58.876535 12655 raft_consensus.cc:2804] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:58.893920 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:58:58.899086 12655 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:58:58.917438 12445 consensus_queue.cc:1048] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260321 02:58:58.924424 12611 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:58:58.888291 12524 consensus_queue.cc:1048] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260321 02:58:58.927513 12460 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:58.934754 11974 raft_consensus.cc:3060] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Advancing to term 2
W20260321 02:58:58.956297 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 94403.1us
I20260321 02:58:58.939855 12655 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:58.958280 12611 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:58.984930 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:58:58.990227 12601 raft_consensus.cc:2804] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:58:58.993013 12601 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:58.993551 12601 raft_consensus.cc:3060] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:58.998610 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:59.000545 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:59.000756 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:59.001565 11233 raft_consensus.cc:2410] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260321 02:58:59.005637 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:59.006378 11303 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:59.008447 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:59.010213 11310 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4"
dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:56630
I20260321 02:58:59.011031 11310 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:58:59.011447 11310 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:59.012343 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:59.013517 11232 raft_consensus.cc:2410] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260321 02:58:59.016749 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.018494 12434 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:59.019170 12434 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:59.014717 12440 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:59.038224 11310 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:59.040243 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:59.040273 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:59.040978 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:59.041730 11230 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:58:59.040412 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.042418 12434 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:59.042977 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:59.043123 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
W20260321 02:58:59.042412 12699 log.cc:927] Time spent T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.080s user 0.001s sys 0.000s
I20260321 02:58:59.045809 11310 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:59.049027 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.050030 11385 raft_consensus.cc:1275] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:59.052636 11974 raft_consensus.cc:2468] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:58:59.020448 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:59.054234 11201 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:58:59.055336 12624 raft_consensus.cc:2804] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:58:59.056053 12624 raft_consensus.cc:697] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:58:59.057502 12624 consensus_queue.cc:237] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:59.061950 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.068413 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.066849 12440 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:59.069157 11376 raft_consensus.cc:2468] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:58:59.025089 12536 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:58:59.072683 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.075058 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.076154 11385 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:58:59.076751 12611 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:59.089046 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:59.090473 11303 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:58:59.103555 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:59.104101 11303 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:58:59.102615 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.104734 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:59.107975 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.109109 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:59.109151 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:59.102421 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:58:59.114867 11275 connection.cc:232] RPC call timeout handler was delayed by -2.1257013370000095s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2089s for a response
W20260321 02:58:59.115422 11275 connection.cc:232] RPC call timeout handler was delayed by -2.1257013370000095s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2089s for a response
I20260321 02:58:59.116634 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:58:59.115927 11275 connection.cc:232] RPC call timeout handler was delayed by -1.6771593090000465s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26320000000000005s for a response
W20260321 02:58:59.120422 11275 connection.cc:232] RPC call timeout handler was delayed by -1.2643771570000126s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1878s for a response
I20260321 02:58:59.121861 11232 raft_consensus.cc:1275] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260321 02:58:59.135260 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 161254us
I20260321 02:58:59.138067 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.138291 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.140154 12702 consensus_queue.cc:1048] T 70e1ea01442f4d13a0f6432ea463c5df P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:58:59.129249 11227 raft_consensus.cc:1240] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:58:59.140905 12601 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:59.143360 12601 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:59.145540 11377 raft_consensus.cc:1275] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:59.146900 12655 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:59.148058 11275 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:58:59.149923 12550 raft_consensus.cc:2764] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260321 02:58:59.150818 11379 raft_consensus.cc:1275] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:58:59.127637 12536 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:58:59.173558 12536 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:58:59.176110 12679 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:58:59.134631 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:58:59.188230 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:58:59.332111 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:58:59.485491 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:59.495205 11379 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:58:59.634958 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:58:59.681828 11302 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.008s sys 0.021s
W20260321 02:58:59.682314 11302 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.562s user 0.008s sys 0.022s
W20260321 02:58:59.761502 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:58:59.770340 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.881458 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:59.881408 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:59.889057 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.892833 11310 raft_consensus.cc:1275] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:59.895183 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.897318 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.898197 11310 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:59.899926 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:59.901816 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.902449 11310 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:59.904146 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.906747 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.909181 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.912060 11305 raft_consensus.cc:1275] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:59.914129 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.914950 11305 raft_consensus.cc:2393] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 2: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:59.916671 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.917631 11305 raft_consensus.cc:2410] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260321 02:58:59.919270 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.922070 11305 raft_consensus.cc:1275] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:58:59.923790 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.924639 11305 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:59.926234 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:58:59.927109 12601 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.779s user 0.034s sys 0.052s
I20260321 02:58:59.927191 11310 raft_consensus.cc:2393] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 3: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
W20260321 02:58:59.927657 12601 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.780s user 0.034s sys 0.052s
I20260321 02:58:59.929481 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.934209 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.935202 11305 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:58:59.938560 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.939888 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.940502 11308 raft_consensus.cc:2468] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 1.
I20260321 02:58:59.941380 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.942075 11308 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:58:59.942910 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.944293 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.946496 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.947237 11308 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:58:59.948137 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.933224 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.935998 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:59.936679 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:59.937574 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:58:59.950779 11307 raft_consensus.cc:2393] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 3: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:58:59.938544 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.963003 11942 leader_election.cc:304] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:58:59.949885 11309 raft_consensus.cc:2376] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3. Re-sending same reply.
W20260321 02:58:59.964385 11303 consensus_meta.cc:287] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.852s user 0.011s sys 0.000s
I20260321 02:58:59.965147 11303 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:58:59.949057 11304 raft_consensus.cc:2376] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3. Re-sending same reply.
I20260321 02:58:59.968353 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.971071 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.971820 11311 raft_consensus.cc:2393] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 2: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:58:59.973980 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.971043 12546 raft_consensus.cc:2804] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:58:59.976027 12546 raft_consensus.cc:493] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:58:59.976406 12546 raft_consensus.cc:3060] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:59.976605 12536 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:58:59.976907 11305 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:58:59.978893 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.979718 11381 raft_consensus.cc:2410] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260321 02:58:59.980304 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.980479 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.981145 11385 raft_consensus.cc:2410] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260321 02:58:59.981711 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.981993 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.982632 11385 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:58:59.982801 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.983412 11377 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:58:59.983487 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:59.978477 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.984485 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.984652 11378 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
I20260321 02:58:59.985127 11383 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
I20260321 02:58:59.985679 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.985968 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.986933 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.987355 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.987929 11382 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:58:59.988035 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:59.988729 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.989013 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:58:59.989271 11382 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:58:59.981069 11381 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:58:59.994685 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:58:59.990718 11383 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:58:59.996147 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:58:59.996721 11385 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:58:59.995677 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:00.001813 11377 raft_consensus.cc:1275] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260321 02:59:00.002321 11377 pending_rounds.cc:85] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5: Aborting all ops after (but not including) 1
I20260321 02:59:00.014982 11377 pending_rounds.cc:107] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 1.2
I20260321 02:59:00.015373 11377 raft_consensus.cc:2917] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Aborting config change with OpId 1.2: Aborted: Op aborted by new leader
I20260321 02:59:00.025496 11377 raft_consensus.cc:2889] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
W20260321 02:59:00.062094 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 44977.5us
I20260321 02:58:59.995651 12679 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:00.003213 12524 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:00.075510 11377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:45622 (request call id 5) took 10352 ms. Trace:
W20260321 02:59:00.094205 11379 consensus_meta.cc:287] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.551s user 0.009s sys 0.001s
I20260321 02:59:00.094628 11379 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:00.081864 12524 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:00.099152 12524 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:00.089149 12679 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:00.100095 12679 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:00.102068 12679 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:00.076675 12655 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:00.076318 11377 rpcz_store.cc:276] 0321 02:58:49.707400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:49.707572 (+ 172us) service_pool.cc:225] Handling call
0321 02:58:49.794339 (+ 86767us) spinlock_profiling.cc:243] Waited 1.71 ms on lock 0x7b58002e3298. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b1522880 00007fa8b150f099 00007fa8b150f362 00007fa8aaa94334
0321 02:58:50.249541 (+455202us) log.cc:844] Serialized 492 byte log entry
0321 02:58:50.322503 (+ 72962us) spinlock_profiling.cc:243] Waited 4.19 ms on lock 0x7b5400155f20. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac4c5522 00007fa8ac4a628a 00007fa8ac4ad133 00007fa8ac4ad0b9 00007fa8ac4ad049 00007fa8ac4ad011 00007fa8ac4ac1dd 0000000000379254 0000000000379099 00007fa8a94b6245
0321 02:58:50.330086 (+ 7583us) spinlock_profiling.cc:243] Waited 7.85 ms on lock 0x7b5400155f20. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac4c5522 00007fa8ac4a628a 00007fa8ac4ad133 00007fa8ac4ad0b9 00007fa8ac4ad049 00007fa8ac4ad011 00007fa8ac4ac1dd 0000000000379254 0000000000379099 00007fa8a94b6245
0321 02:58:50.331062 (+ 976us) spinlock_profiling.cc:243] Waited 2.3 ms on lock 0x7b5400155f20. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac4ccf06 00007fa8ac4c687a 00007fa8ac4c5af4 00007fa8ac4a628a 00007fa8ac4ad133 00007fa8ac4ad0b9 00007fa8ac4ad049 00007fa8ac4ad011 00007fa8ac4ac1dd 0000000000379254
0321 02:59:00.026214 (+9695152us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1390,"raft.run_cpu_time_us":4177,"raft.run_wall_time_us":45402,"spinlock_wait_cycles":36927232,"thread_start_us":24816,"threads_started":1,"wal-append.queue_time_us":72000,"wal-append.run_cpu_time_us":3591,"wal-append.run_wall_time_us":1140150}
I20260321 02:59:00.111013 12546 raft_consensus.cc:515] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:00.121490 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:00.117964 12536 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:00.125406 11306 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:00.129040 11385 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
W20260321 02:59:00.127722 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 155787us
I20260321 02:59:00.134117 12536 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:00.136788 12536 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:00.138258 12546 leader_election.cc:290] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:00.145648 11975 raft_consensus.cc:1240] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:00.145892 11974 raft_consensus.cc:1240] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:00.151444 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:00.154258 12679 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:00.154943 12679 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:00.163606 11976 raft_consensus.cc:1240] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:00.165329 12713 consensus_queue.cc:1059] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:59:00.166477 12713 raft_consensus.cc:3055] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:59:00.166822 12713 raft_consensus.cc:740] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:00.172749 12679 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:00.174046 12550 consensus_queue.cc:1059] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:59:00.181128 12550 consensus_queue.cc:1059] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:59:00.199736 12434 raft_consensus.cc:3055] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:59:00.200363 12434 raft_consensus.cc:740] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
W20260321 02:59:00.201622 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 73940.5us
I20260321 02:59:00.203891 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.205156 11227 raft_consensus.cc:2393] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 4: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:00.206632 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.207798 11234 raft_consensus.cc:2393] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 4: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:00.201524 12434 consensus_queue.cc:260] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:00.209372 11349 connection.cc:232] RPC call timeout handler was delayed by -1.1089604820000432s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.215s for a response
I20260321 02:59:00.210554 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.212078 12524 consensus_queue.cc:1059] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:00.167804 12713 consensus_queue.cc:260] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:00.220118 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:00.220969 11232 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:00.221940 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 68146.4us
I20260321 02:59:00.224383 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.225282 11230 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:00.228029 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.229409 11972 raft_consensus.cc:1275] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260321 02:59:00.234997 11974 raft_consensus.cc:1240] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:00.233897 12679 consensus_queue.cc:1048] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260321 02:59:00.236011 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:00.236792 11230 raft_consensus.cc:3055] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:59:00.237146 11230 raft_consensus.cc:740] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:00.238211 11230 consensus_queue.cc:260] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:00.239414 11230 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:00.241842 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:00.244606 12679 consensus_queue.cc:1059] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:59:00.245481 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.245781 12720 raft_consensus.cc:3055] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:59:00.246260 12720 raft_consensus.cc:740] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:00.247248 12720 consensus_queue.cc:260] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:00.252477 12720 raft_consensus.cc:3060] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
W20260321 02:59:00.272447 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 96043.8us
I20260321 02:59:00.283550 12524 raft_consensus.cc:3055] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:00.284081 12524 raft_consensus.cc:740] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:00.284291 11230 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
I20260321 02:59:00.285059 12524 consensus_queue.cc:260] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:00.286368 12524 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:00.286672 11232 raft_consensus.cc:2410] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260321 02:59:00.286391 12536 raft_consensus.cc:3055] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Stepping down as leader of term 1
W20260321 02:59:00.294888 11182 auto_rebalancer.cc:274] failed to send replica move request: Aborted: Op aborted by new leader
I20260321 02:59:00.294981 12536 raft_consensus.cc:740] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:00.296275 12536 consensus_queue.cc:260] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:00.297844 12536 raft_consensus.cc:3060] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
W20260321 02:59:00.300153 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 100882us
I20260321 02:59:00.301407 12434 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
W20260321 02:59:00.307855 11182 auto_rebalancer.cc:791] Removing replace marker failed: Replica fef7f8cc55b043039131bd9b2229f0a5 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
W20260321 02:59:00.308815 11182 auto_rebalancer.cc:799] Could not move replica: Incomplete: tablet c5fc517dbbd646bfa0d307d4e42556ba, TS 83b12b883a7f4129a87dadf5a78f08da -> TS 3d8d3176d6ec407ea2a965556c2e2c56 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:00.309319 11182 auto_rebalancer.cc:284] scheduled replica move failed to complete: Incomplete: tablet c5fc517dbbd646bfa0d307d4e42556ba, TS 83b12b883a7f4129a87dadf5a78f08da -> TS 3d8d3176d6ec407ea2a965556c2e2c56 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:00.311328 12709 log.cc:927] Time spent T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.239s user 0.000s sys 0.000s
W20260321 02:59:00.326140 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:00.332914 11302 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1201) took 1328 ms. Trace:
I20260321 02:59:00.333503 11302 rpcz_store.cc:276] 0321 02:58:59.004361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:59.008942 (+ 4581us) service_pool.cc:225] Handling call
0321 02:58:59.009273 (+ 331us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:58:59.009802 (+ 529us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:58:59.009840 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:58:59.079268 (+ 69428us) spinlock_profiling.cc:243] Waited 1.55 ms on lock 0x7b2c00010da8. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8aae48c81 00007fa8aae585e2 00007fa8ac587de1 00007fa8ac54c81a 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22
0321 02:58:59.097112 (+ 17844us) log.cc:844] Serialized 26 byte log entry
0321 02:58:59.682757 (+585645us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:58:59.682809 (+ 52us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:59.683106 (+ 297us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:00.310629 (+627523us) spinlock_profiling.cc:243] Waited 4.18 ms on lock 0x7b440002304c. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93c2dc5 00007fa8a93c37e2 00007fa8ac0f757b 00007fa8ac0f2e4f 00007fa8ac0e6c0e 00007fa8ac0e5f56 00007fa8ac0e5b78 00007fa8ac0ef5c1 00007fa8ac0ef579 00007fa8ac0ef509
0321 02:59:00.332491 (+ 21862us) raft_consensus.cc:1685] finished
0321 02:59:00.332526 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:00.332725 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8398,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":104,"spinlock_wait_cycles":13175296,"thread_start_us":562362,"threads_started":1,"wal-append.queue_time_us":855299}
I20260321 02:59:00.344672 11380 raft_consensus.cc:1275] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:59:00.346693 11377 raft_consensus.cc:1275] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260321 02:59:00.347216 11377 pending_rounds.cc:85] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5: Aborting all ops after (but not including) 1
I20260321 02:59:00.347502 11377 pending_rounds.cc:107] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260321 02:59:00.347781 11377 raft_consensus.cc:2889] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260321 02:59:00.368928 12711 log.cc:927] Time spent T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.272s user 0.003s sys 0.000s
I20260321 02:59:00.370635 11384 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1207) took 1232 ms. Trace:
I20260321 02:59:00.371198 11384 rpcz_store.cc:276] 0321 02:58:59.138111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:59.138330 (+ 219us) service_pool.cc:225] Handling call
0321 02:58:59.138711 (+ 381us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:58:59.139097 (+ 386us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:58:59.139125 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:58:59.139674 (+ 549us) log.cc:844] Serialized 26 byte log entry
0321 02:58:59.478474 (+338800us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:58:59.479046 (+ 572us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:58:59.479204 (+ 158us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:00.368282 (+889078us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b440002304c. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93c2dc5 00007fa8a93c37e2 00007fa8ac0f757b 00007fa8ac0f2e4f 00007fa8ac0e6c0e 00007fa8ac0e5f56 00007fa8ac0e5b78 00007fa8ac0ef5c1 00007fa8ac0ef579 00007fa8ac0ef509
0321 02:59:00.370088 (+ 1806us) raft_consensus.cc:1685] finished
0321 02:59:00.370117 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:00.370423 (+ 306us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39893888,"thread_start_us":298895,"threads_started":1,"wal-append.queue_time_us":941869}
I20260321 02:59:00.392146 12713 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
W20260321 02:59:00.410689 11200 connection.cc:232] RPC call timeout handler was delayed by -2.0248048689999223s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23199999999999998s for a response
I20260321 02:59:00.439319 12715 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:00.444231 11230 raft_consensus.cc:1275] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:59:00.444355 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.444509 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:00.445166 11227 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:00.445294 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.447258 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.447557 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.449839 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.450534 11230 raft_consensus.cc:2468] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:59:00.450822 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.436307 12720 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:00.451676 11234 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 1.
I20260321 02:59:00.441453 12715 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:00.451687 12720 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:00.454274 12715 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:00.454857 12715 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:00.455319 12715 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:00.457533 12720 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:00.457907 11232 consensus_queue.cc:237] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:00.457604 12715 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:00.450024 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.464232 11229 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:59:00.472064 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.472663 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.472913 11229 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:59:00.473891 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.476405 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.476980 11230 raft_consensus.cc:2393] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 2: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:00.477883 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.480208 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.475291 11227 raft_consensus.cc:2468] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:59:00.483830 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:00.504174 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:00.508047 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.517130 11230 raft_consensus.cc:1240] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:00.520278 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.521179 11229 raft_consensus.cc:2468] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:59:00.520404 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.521641 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:00.524619 11234 raft_consensus.cc:2376] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3. Re-sending same reply.
I20260321 02:59:00.528795 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.529488 11234 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:59:00.532538 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.534282 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.534879 11229 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:59:00.531906 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.536967 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.538365 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.538929 11200 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:00.539230 11234 raft_consensus.cc:2393] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:00.540117 12715 raft_consensus.cc:2749] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:00.540544 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:00.541478 11229 raft_consensus.cc:2393] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:00.546041 11200 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:00.547369 12720 raft_consensus.cc:2804] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:00.548084 12720 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:00.548493 12720 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:00.549594 11200 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:00.550449 12715 raft_consensus.cc:2804] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:00.550827 12715 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:00.551174 12715 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:00.555919 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:00.561767 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:00.563921 12556 consensus_queue.cc:1048] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:59:00.568351 12720 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:00.571370 12720 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:00.573906 12715 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:00.578253 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 367353us
I20260321 02:59:00.576697 12715 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:00.595482 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 46010.2us
I20260321 02:59:00.630654 11200 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:00.634068 12715 raft_consensus.cc:2804] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:00.634785 12715 raft_consensus.cc:697] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:00.635993 12715 consensus_queue.cc:237] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:00.659485 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:00.682185 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 66210.5us
I20260321 02:59:00.683535 11200 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:00.714625 11349 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:00.715534 12725 consensus_queue.cc:1059] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:00.716118 12728 consensus_queue.cc:1048] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:00.717192 12725 raft_consensus.cc:2804] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:00.715876 11349 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:00.765048 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 49276.2us
W20260321 02:59:00.765897 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 47136.9us
I20260321 02:59:00.766816 11200 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:00.767071 12740 consensus_queue.cc:1048] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:00.768198 11349 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:00.769188 12742 raft_consensus.cc:2804] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:00.769636 12742 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:00.770025 12742 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:00.770511 12740 raft_consensus.cc:2804] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:00.771315 12740 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:00.772581 12740 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:00.744180 12725 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:00.774482 12725 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:00.757848 12720 raft_consensus.cc:2804] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:00.780974 11349 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:00.781827 12739 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:00.782718 11349 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:00.789179 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:00.782229 12739 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:00.779561 12742 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:00.768069 12724 consensus_queue.cc:1048] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260321 02:59:00.796346 12739 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:00.796931 12747 raft_consensus.cc:2749] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:00.800004 12749 raft_consensus.cc:2804] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:00.822530 12749 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:00.822968 12749 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:00.807117 12738 raft_consensus.cc:3055] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:00.828012 12738 raft_consensus.cc:740] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:00.829262 12738 consensus_queue.cc:260] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:00.835628 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:00.836442 11384 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:00.836519 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:00.804785 12720 raft_consensus.cc:697] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:00.837304 11380 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:59:00.840276 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:00.842353 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:00.843180 11380 raft_consensus.cc:2468] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:59:00.845402 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:00.847189 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:00.848383 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:00.848702 12742 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:00.849156 11376 raft_consensus.cc:3060] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:00.854696 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:00.855603 12740 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:00.856245 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:00.865748 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:00.866909 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 67588.1us
I20260321 02:59:00.867373 11380 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:00.877863 11384 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:59:00.881104 12739 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:00.882987 12740 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:00.885878 11974 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s user 0.037s sys 0.057s
I20260321 02:59:00.887311 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:00.887853 12739 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:00.888048 11379 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:59:00.826769 12725 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:00.892966 11376 raft_consensus.cc:2468] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
W20260321 02:59:00.886438 11974 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.646s user 0.037s sys 0.057s
I20260321 02:59:00.898047 11376 raft_consensus.cc:1275] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:59:00.899585 12720 consensus_queue.cc:237] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:00.906359 11379 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "42855a31af244db9b41f42c30304de34"
dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
mode: GRACEFUL
new_leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
from {username='slave'} at 127.0.0.1:44010
I20260321 02:59:00.907272 11379 raft_consensus.cc:606] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Received request to transfer leadership to TS 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:00.907642 11379 raft_consensus.cc:614] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260321 02:59:00.908046 11350 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 64340.8us
W20260321 02:59:00.922953 11276 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.649s user 0.022s sys 0.002s
W20260321 02:59:00.923466 11276 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.650s user 0.024s sys 0.002s
I20260321 02:59:00.923647 11384 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda"
dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:44010
I20260321 02:59:00.924342 11384 raft_consensus.cc:606] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:00.904772 12725 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:00.903009 12524 consensus_meta.cc:287] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.616s user 0.013s sys 0.007s
W20260321 02:59:00.923725 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 650288us
I20260321 02:59:00.923039 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:00.885918 12749 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:00.932008 11133 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42514 (request call id 208) took 6501 ms. Trace:
W20260321 02:59:00.948668 11349 leader_election.cc:336] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.619s (SENT)
I20260321 02:59:00.949227 11349 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:00.950593 12738 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:00.951782 11275 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.650s user 0.009s sys 0.015s
W20260321 02:59:00.952382 11275 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.651s user 0.009s sys 0.015s
W20260321 02:59:00.952646 11349 leader_election.cc:336] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.661s (SENT)
W20260321 02:59:00.952878 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 651409us
W20260321 02:59:00.957722 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 165724us
I20260321 02:59:00.955876 12496 raft_consensus.cc:2749] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:00.959187 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:00.960947 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:00.964844 11275 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
W20260321 02:59:00.966018 11349 leader_election.cc:336] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.786s (SENT)
I20260321 02:59:00.966753 11200 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:00.966639 11349 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:00.970145 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:00.970577 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:00.970876 11302 raft_consensus.cc:2468] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:59:00.972687 11376 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b"
dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
mode: GRACEFUL
new_leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
from {username='slave'} at 127.0.0.1:44010
I20260321 02:59:00.973143 11376 raft_consensus.cc:606] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Received request to transfer leadership to TS 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:00.973362 11376 raft_consensus.cc:614] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:00.979082 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:00.979782 12724 raft_consensus.cc:2749] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:00.979202 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:00.980484 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:00.976836 12723 raft_consensus.cc:2804] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:00.981149 11309 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:00.981477 12723 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:00.981971 12723 raft_consensus.cc:3060] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:00.982662 12722 consensus_queue.cc:1059] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:59:00.968219 12720 raft_consensus.cc:2749] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:00.983690 12434 raft_consensus.cc:3055] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:59:00.984131 12434 raft_consensus.cc:740] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:00.984644 12720 consensus_queue.cc:1048] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260321 02:59:00.985435 11349 leader_election.cc:336] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.789s (SENT)
W20260321 02:59:00.986048 11349 leader_election.cc:336] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Timed out: RequestConsensusVote RPC to 127.9.213.1:36717 timed out after 1.789s (SENT)
I20260321 02:59:00.986416 11349 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:00.985041 12434 consensus_queue.cc:260] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:00.987182 12724 raft_consensus.cc:2749] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:00.987378 11200 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:00.987371 11130 catalog_manager.cc:5671] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:00.989519 12715 raft_consensus.cc:2804] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:59:00.989895 12715 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:00.990255 12715 raft_consensus.cc:3060] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:00.990479 11200 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:00.990108 11130 catalog_manager.cc:5671] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:00.991832 12757 raft_consensus.cc:2749] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:00.992257 11130 catalog_manager.cc:5671] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:59:00.993867 11130 catalog_manager.cc:5671] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 3, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4). New cstate: current_term: 3 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:59:00.994743 12757 consensus_queue.cc:1048] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:59:00.995671 11130 catalog_manager.cc:5671] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:59:00.997254 12749 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:00.997635 12757 consensus_queue.cc:1048] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:00.997524 11130 catalog_manager.cc:5671] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:59:00.998713 12715 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:00.987512 12434 raft_consensus.cc:3060] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:00.999533 11130 catalog_manager.cc:5671] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:59:01.001044 12757 consensus_queue.cc:1048] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:59:01.001499 11130 catalog_manager.cc:5671] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:59:01.003223 11130 catalog_manager.cc:5671] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:59:01.003559 12720 consensus_queue.cc:1048] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:59:01.005097 11130 catalog_manager.cc:5671] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:01.006865 11130 catalog_manager.cc:5671] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:59:01.007556 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:01.009550 11275 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:01.010879 12536 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:01.010329 11130 catalog_manager.cc:5671] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 3, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 3 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:01.011307 12536 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:01.011799 12536 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:01.011884 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:01.012907 11130 catalog_manager.cc:5671] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:01.014045 11275 leader_election.cc:304] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:01.006229 11200 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:01.016214 11200 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:01.018245 11200 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:01.018930 11130 catalog_manager.cc:5671] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:59:01.020184 11200 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:01.021242 11130 catalog_manager.cc:5671] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:59:01.023277 12720 raft_consensus.cc:2749] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:01.024705 12730 raft_consensus.cc:2749] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:01.024593 11130 catalog_manager.cc:5671] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 2 to 3, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4). New cstate: current_term: 3 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:59:01.023851 12752 raft_consensus.cc:2804] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:59:01.028782 12752 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:01.029577 11233 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:01.029834 12752 raft_consensus.cc:3060] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:01.030331 11233 raft_consensus.cc:606] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:01.024540 12757 raft_consensus.cc:2804] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:59:01.033178 12757 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:01.033577 12757 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:01.033869 11233 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:01.034467 11233 raft_consensus.cc:606] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:01.035079 11233 raft_consensus.cc:614] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:01.037863 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:01.038215 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:01.039846 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260321 02:59:01.041069 11275 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:01.037904 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:01.041584 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:59:00.951655 11133 rpcz_store.cc:276] 0321 02:58:54.430938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:58:54.431098 (+ 160us) service_pool.cc:225] Handling call
0321 02:58:54.832282 (+401184us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b54002d69d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b6470 00007fa8ac55531a 00007fa8b22e365c 00007fa8b23dd5de 00007fa8b08d64a4 00007fa8b08d6432 00007fa8b08d6361 00007fa8b08d62dc 00007fa8b08d5592 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:00.931892 (+6099610us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:58:58.493643 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:58:58.494342 (+ 699us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:58:58.494398 (+ 56us) write_op.cc:435] Acquired schema lock
0321 02:58:58.494435 (+ 37us) tablet.cc:661] Decoding operations
0321 02:58:58.528982 (+ 34547us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:58:58.590664 (+ 61682us) write_op.cc:641] Partition lock acquired for write op
0321 02:58:58.590711 (+ 47us) tablet.cc:684] Acquiring locks for 1 operations
0321 02:58:58.591046 (+ 335us) tablet.cc:700] Row locks acquired
0321 02:58:58.591080 (+ 34us) write_op.cc:260] PREPARE: finished
0321 02:58:58.591262 (+ 182us) write_op.cc:270] Start()
0321 02:58:58.591415 (+ 153us) write_op.cc:276] Timestamp: P: 1774061938591192 usec, L: 0
0321 02:58:58.591458 (+ 43us) op_driver.cc:348] REPLICATION: starting
0321 02:58:58.592387 (+ 929us) log.cc:844] Serialized 535 byte log entry
0321 02:58:59.101828 (+509441us) op_driver.cc:464] REPLICATION: finished
0321 02:59:00.092822 (+990994us) write_op.cc:301] APPLY: starting
0321 02:59:00.095474 (+ 2652us) tablet.cc:1366] starting BulkCheckPresence
0321 02:59:00.095620 (+ 146us) tablet.cc:1369] finished BulkCheckPresence
0321 02:59:00.095758 (+ 138us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:59:00.096093 (+ 335us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:59:00.096368 (+ 275us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0321 02:59:00.096434 (+ 66us) write_op.cc:312] APPLY: finished
0321 02:59:00.096788 (+ 354us) log.cc:844] Serialized 23 byte log entry
0321 02:59:00.341791 (+245003us) spinlock_profiling.cc:243] Waited 9.01 ms on lock 0x7b440002304c. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93c2dc5 00007fa8a93c37e2 00007fa8ac0f757b 00007fa8ac0f2e4f 00007fa8a9498fbc 00007fa8b2276b19 00007fa8a94b546b 00007fa8a94b370f 00007fa8a94b59d2 00007fa8ac0e55ab
0321 02:59:00.394099 (+ 52308us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:00.394202 (+ 103us) write_op.cc:454] Released schema lock
0321 02:59:00.394288 (+ 86us) write_op.cc:341] FINISH: Updating metrics
0321 02:59:00.394881 (+ 593us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:00.394925 (+ 44us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":690340,"reported_tablets":3,"spinlock_wait_cycles":364321664,"child_traces":[["op",{"apply.queue_time_us":972729,"apply.run_cpu_time_us":13195,"apply.run_wall_time_us":306393,"num_ops":1,"prepare.queue_time_us":32447,"prepare.run_cpu_time_us":34419,"prepare.run_wall_time_us":242534,"raft.queue_time_us":214604,"raft.run_cpu_time_us":25247,"raft.run_wall_time_us":436958,"replication_time_us":506909,"spinlock_wait_cycles":20719488,"thread_start_us":974690,"threads_started":5,"wal-append.queue_time_us":459421,"wal-append.run_cpu_time_us":56237,"wal-append.run_wall_time_us":1175914}]]}
I20260321 02:59:01.043380 11200 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:01.044111 11275 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:01.046087 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:01.046932 11306 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:01.048249 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:01.048696 12715 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:01.050150 12722 raft_consensus.cc:2749] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:59:01.045082 12730 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:01.052203 12730 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:01.052884 12730 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:01.025343 12434 consensus_queue.cc:1048] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260321 02:59:01.056278 12736 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:01.056788 12736 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:01.045262 11200 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:01.058648 12715 raft_consensus.cc:2804] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:59:01.059171 12715 raft_consensus.cc:697] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:01.059933 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:01.061255 11309 raft_consensus.cc:1275] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:59:01.057137 12752 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:01.060170 12715 consensus_queue.cc:237] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:01.064286 12752 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:01.066004 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:01.076710 11309 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:01.076570 12753 raft_consensus.cc:2749] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:01.057165 12736 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:01.057161 11311 raft_consensus.cc:2468] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:59:01.068755 11306 raft_consensus.cc:1240] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:01.069041 11200 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:01.081362 11200 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:01.082195 12749 raft_consensus.cc:2804] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:59:01.082633 12749 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:01.083076 12749 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:01.083854 12752 raft_consensus.cc:2804] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:59:01.086226 12752 raft_consensus.cc:697] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
W20260321 02:59:01.089951 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:01.017273 12723 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:01.098237 12723 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:01.105592 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 61553.1us
I20260321 02:59:01.108343 12753 consensus_queue.cc:1048] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:01.123342 11200 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:01.126658 11302 raft_consensus.cc:2468] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:59:01.123406 11275 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:01.141074 12679 raft_consensus.cc:2749] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:01.142009 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.146376 12536 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:01.152280 11942 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:01.153297 12749 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:01.156358 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:01.097718 12752 consensus_queue.cc:237] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:01.158288 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.158965 11233 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:01.162426 12556 raft_consensus.cc:2749] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:59:01.164284 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.169260 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:01.170928 11275 connection.cc:232] RPC call timeout handler was delayed by -1.0491101549998803s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19s for a response
I20260321 02:59:01.170439 11302 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:01.173800 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:01.172276 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.163620 12536 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:01.173084 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.174517 11376 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:01.179440 11377 raft_consensus.cc:1240] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:01.180298 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:01.183635 11377 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:01.183997 11311 raft_consensus.cc:1275] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260321 02:59:01.185810 11311 raft_consensus.cc:1240] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 2. Current term is 3. Ops: []
I20260321 02:59:01.187112 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:01.155138 12736 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:01.187680 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:01.188678 11383 raft_consensus.cc:2393] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 2: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:01.189246 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:01.190474 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:01.149099 12722 raft_consensus.cc:2804] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:59:01.190582 12757 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:01.192312 12722 raft_consensus.cc:697] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 2 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:01.193536 12757 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:01.193282 12722 consensus_queue.cc:237] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:01.195248 12749 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:01.195436 11232 raft_consensus.cc:1240] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:01.197237 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:01.201274 12730 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:01.202225 11236 raft_consensus.cc:1240] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260321 02:59:01.188613 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:01.189783 12736 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:01.207634 12738 consensus_queue.cc:1048] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:01.208897 11306 raft_consensus.cc:2393] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 4: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:01.211318 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:01.212199 11232 raft_consensus.cc:3060] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:01.214203 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:01.214941 11376 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:01.220573 11302 raft_consensus.cc:1275] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:59:01.223651 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:01.224370 11311 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:01.234498 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:01.236539 12724 consensus_queue.cc:1059] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:01.236991 12730 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:01.237417 11349 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:01.237550 12496 raft_consensus.cc:3055] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:01.239235 12496 raft_consensus.cc:740] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:01.240136 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:01.240302 12496 consensus_queue.cc:260] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:01.241432 12496 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:01.242329 12724 raft_consensus.cc:2804] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:01.242789 12724 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:01.243158 12724 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:01.244571 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:01.245525 11349 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:01.250598 11377 raft_consensus.cc:1275] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:59:01.253332 11232 raft_consensus.cc:2468] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:01.258637 12738 raft_consensus.cc:2804] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:01.259738 12738 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:01.268247 12738 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:01.276378 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:01.278512 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:01.293007 11311 raft_consensus.cc:1275] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260321 02:59:01.292868 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:01.295871 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:01.299088 11309 raft_consensus.cc:2393] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 4: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:01.299685 11379 raft_consensus.cc:1240] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260321 02:59:01.276993 12724 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:01.301709 11380 raft_consensus.cc:1240] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260321 02:59:01.302086 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:01.302646 11306 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:01.302862 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:01.305742 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:01.301748 11384 raft_consensus.cc:1275] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260321 02:59:01.312891 11376 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:01.310006 12724 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:01.317057 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:01.317759 11310 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:01.317934 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:01.321858 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:01.323244 11384 raft_consensus.cc:1275] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260321 02:59:01.326100 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:01.327257 11384 raft_consensus.cc:1240] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:01.331009 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:01.335215 11306 raft_consensus.cc:2468] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:01.331682 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:01.341663 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 163750us
I20260321 02:59:01.351223 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:01.352597 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.364451 11233 raft_consensus.cc:1240] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 2. Current term is 3. Ops: []
I20260321 02:59:01.376564 11275 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:01.374241 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.380967 11233 raft_consensus.cc:2393] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:01.384634 11275 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:01.379518 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:01.386863 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.388265 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.362044 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.388029 11132 catalog_manager.cc:5671] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:01.389926 11310 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:01.396620 11132 catalog_manager.cc:5671] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 3 to 4, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 4 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:01.399093 11130 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42530 (request call id 214) took 1126 ms. Trace:
I20260321 02:59:01.399710 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.380429 11380 raft_consensus.cc:1240] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:01.402381 11132 catalog_manager.cc:5671] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 1 to 2, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 2 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:01.406875 11132 catalog_manager.cc:5671] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:01.411099 11275 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:01.400903 11130 rpcz_store.cc:276] 0321 02:59:00.265879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:00.267641 (+ 1762us) service_pool.cc:225] Handling call
0321 02:59:01.392552 (+1124911us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:59:01.181772 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:59:01.182400 (+ 628us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:59:01.182434 (+ 34us) write_op.cc:435] Acquired schema lock
0321 02:59:01.182453 (+ 19us) tablet.cc:661] Decoding operations
0321 02:59:01.183871 (+ 1418us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:59:01.183992 (+ 121us) write_op.cc:641] Partition lock acquired for write op
0321 02:59:01.184021 (+ 29us) tablet.cc:684] Acquiring locks for 16 operations
0321 02:59:01.185240 (+ 1219us) tablet.cc:700] Row locks acquired
0321 02:59:01.185262 (+ 22us) write_op.cc:260] PREPARE: finished
0321 02:59:01.185406 (+ 144us) write_op.cc:270] Start()
0321 02:59:01.185534 (+ 128us) write_op.cc:276] Timestamp: P: 1774061941185358 usec, L: 0
0321 02:59:01.185557 (+ 23us) op_driver.cc:348] REPLICATION: starting
0321 02:59:01.186361 (+ 804us) log.cc:844] Serialized 6256 byte log entry
0321 02:59:01.229075 (+ 42714us) op_driver.cc:464] REPLICATION: finished
0321 02:59:01.281110 (+ 52035us) write_op.cc:301] APPLY: starting
0321 02:59:01.281203 (+ 93us) tablet.cc:1366] starting BulkCheckPresence
0321 02:59:01.281575 (+ 372us) tablet.cc:1369] finished BulkCheckPresence
0321 02:59:01.281858 (+ 283us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:59:01.284240 (+ 2382us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:59:01.284542 (+ 302us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=16
0321 02:59:01.284595 (+ 53us) write_op.cc:312] APPLY: finished
0321 02:59:01.284996 (+ 401us) log.cc:844] Serialized 113 byte log entry
0321 02:59:01.285179 (+ 183us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:01.285342 (+ 163us) write_op.cc:454] Released schema lock
0321 02:59:01.285396 (+ 54us) write_op.cc:341] FINISH: Updating metrics
0321 02:59:01.285794 (+ 398us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:01.285825 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":107122,"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":51776,"apply.run_cpu_time_us":4280,"apply.run_wall_time_us":4661,"num_ops":16,"prepare.queue_time_us":110988,"prepare.run_cpu_time_us":4940,"prepare.run_wall_time_us":4939,"replication_time_us":43370,"thread_start_us":155506,"threads_started":2}]]}
I20260321 02:59:01.394201 12730 raft_consensus.cc:2749] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:01.413913 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:01.416491 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:01.419344 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.420552 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.421281 11236 raft_consensus.cc:2393] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 2: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:01.422106 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.423249 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.412137 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:01.401714 12536 raft_consensus.cc:2749] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260321 02:59:01.426991 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:01.412302 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:01.418063 11132 catalog_manager.cc:5671] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:01.431747 11232 raft_consensus.cc:2468] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 2.
I20260321 02:59:01.430083 11275 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:01.433312 12722 raft_consensus.cc:2749] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:01.433604 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.432020 11228 raft_consensus.cc:2376] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2. Re-sending same reply.
I20260321 02:59:01.435227 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.437952 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.438450 11275 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:01.439406 12722 raft_consensus.cc:2749] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 2 LEADER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:01.441593 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.442402 11275 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:01.427424 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.443552 12434 raft_consensus.cc:2749] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:01.443182 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.439672 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:01.445312 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.445305 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.447582 11232 raft_consensus.cc:2393] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
W20260321 02:59:01.444193 11257 debug-util.cc:398] Leaking SignalData structure 0x7b08002bae20 after lost signal to thread 10071
W20260321 02:59:01.450201 11257 debug-util.cc:398] Leaking SignalData structure 0x7b080010e6c0 after lost signal to thread 11168
W20260321 02:59:01.452519 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:01.451818 11257 debug-util.cc:398] Leaking SignalData structure 0x7b08000c6600 after lost signal to thread 11260
W20260321 02:59:01.453832 11257 debug-util.cc:398] Leaking SignalData structure 0x7b08000c65a0 after lost signal to thread 11261
W20260321 02:59:01.454800 11257 debug-util.cc:398] Leaking SignalData structure 0x7b080013c100 after lost signal to thread 11335
W20260321 02:59:01.455865 11257 debug-util.cc:398] Leaking SignalData structure 0x7b080037fc60 after lost signal to thread 11409
I20260321 02:59:01.431113 12761 raft_consensus.cc:2749] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:01.447407 11132 catalog_manager.cc:5671] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:01.481750 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.486207 11275 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:01.489080 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.489724 11236 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:59:01.488456 11132 catalog_manager.cc:5671] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 1 to 2, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 2 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:01.455338 12730 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:01.506232 11200 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:01.502115 11275 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:01.509017 11349 leader_election.cc:336] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.525s (SENT)
I20260321 02:59:01.511360 11349 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:01.508843 11132 catalog_manager.cc:5671] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 1 to 3, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 3 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:01.513046 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:01.493811 12722 raft_consensus.cc:2749] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:01.515187 11236 raft_consensus.cc:1240] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:01.515182 11200 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:01.532843 11228 raft_consensus.cc:1240] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:01.546908 11275 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
W20260321 02:59:01.521984 12776 log.cc:927] Time spent T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.122s user 0.001s sys 0.000s
I20260321 02:59:01.514444 12753 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:01.576634 12753 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:01.577224 12753 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:01.510166 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:01.561214 11257 debug-util.cc:398] Leaking SignalData structure 0x7b08003824a0 after lost signal to thread 12004
W20260321 02:59:01.597815 11257 debug-util.cc:398] Leaking SignalData structure 0x7b08003826a0 after lost signal to thread 12422
I20260321 02:59:01.533406 11974 raft_consensus.cc:1240] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:01.564316 12752 raft_consensus.cc:2749] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:01.564891 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:01.575387 12434 raft_consensus.cc:2804] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:01.544059 12496 raft_consensus.cc:2749] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260321 02:59:01.587580 11349 leader_election.cc:336] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.604s (SENT)
W20260321 02:59:01.598598 11257 debug-util.cc:398] Leaking SignalData structure 0x7b0800382500 after lost signal to thread 12685
I20260321 02:59:01.620270 11349 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:01.515041 12730 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:01.605553 11232 raft_consensus.cc:1240] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:01.562698 11228 raft_consensus.cc:1240] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: []
W20260321 02:59:01.631412 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 43907.7us
I20260321 02:59:01.617776 12434 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:01.641052 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 94200.6us
I20260321 02:59:01.645320 11275 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:01.625830 12724 raft_consensus.cc:2749] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:01.624228 11227 raft_consensus.cc:2376] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 2. Re-sending same reply.
I20260321 02:59:01.625268 12738 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:01.656714 12752 consensus_queue.cc:1059] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260321 02:59:01.658130 12752 raft_consensus.cc:3055] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:01.658545 12752 raft_consensus.cc:740] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:01.657063 11385 raft_consensus.cc:1240] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:01.675124 12730 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:01.673123 12738 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:01.647429 12434 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:01.679569 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:01.687160 11974 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
I20260321 02:59:01.689088 11276 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:01.692737 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 177620us
I20260321 02:59:01.695186 11376 raft_consensus.cc:1240] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 2. Current term is 3. Ops: []
W20260321 02:59:01.702598 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 57412us
I20260321 02:59:01.694346 12723 raft_consensus.cc:2804] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:01.659654 12752 consensus_queue.cc:260] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:01.702332 12761 raft_consensus.cc:2749] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:01.704108 12722 raft_consensus.cc:2804] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:01.703112 12655 consensus_queue.cc:1059] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260321 02:59:01.704396 11275 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:01.742053 11275 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:01.701581 11200 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:01.751354 11275 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:01.751675 11200 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:01.753450 11275 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:01.708355 12752 raft_consensus.cc:3060] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:01.761302 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:01.705418 12723 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:01.718691 12722 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:01.725200 12655 raft_consensus.cc:3055] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:59:01.752381 12783 raft_consensus.cc:2804] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:01.751143 12536 raft_consensus.cc:2749] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:01.705806 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:01.750962 12785 raft_consensus.cc:2804] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:01.718394 12753 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:01.767976 12723 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:01.759346 12679 consensus_queue.cc:1048] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260321 02:59:01.776297 12722 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:01.765693 12784 raft_consensus.cc:2804] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:01.780201 12785 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:01.777729 12655 raft_consensus.cc:740] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:01.795158 11236 raft_consensus.cc:1240] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260321 02:59:01.779237 12749 raft_consensus.cc:2804] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:01.795104 12701 consensus_queue.cc:1059] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260321 02:59:01.797415 12784 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:01.778723 12434 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:01.778005 12783 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:01.793715 12536 raft_consensus.cc:2749] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260321 02:59:01.811801 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 60166.2us
I20260321 02:59:01.815562 11200 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:01.817169 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 63767us
I20260321 02:59:01.818832 11275 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:01.822463 11349 leader_election.cc:336] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.835s (SENT)
I20260321 02:59:01.822986 11349 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:01.812564 12783 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:01.810218 12784 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:01.809749 12749 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:01.802208 12785 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:01.823272 11275 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:01.826582 12655 consensus_queue.cc:260] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:01.827898 12655 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:01.826318 11275 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:01.830227 12761 raft_consensus.cc:2804] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:59:01.830633 12761 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:01.831144 12761 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:01.832849 12753 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:01.833052 11275 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:01.833564 12753 consensus_queue.cc:1048] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:59:01.834003 12730 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:01.834478 12722 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:01.820142 12715 raft_consensus.cc:2804] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:01.836938 11275 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
W20260321 02:59:01.838747 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:01.821862 12536 raft_consensus.cc:2749] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260321 02:59:01.860852 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 76705us
I20260321 02:59:01.853355 12715 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:01.865588 12715 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:01.865877 12782 log.cc:927] Time spent T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
W20260321 02:59:01.869289 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 46861.3us
W20260321 02:59:01.868077 12791 log.cc:927] Time spent T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.099s user 0.001s sys 0.000s
I20260321 02:59:01.835690 12753 consensus_queue.cc:1059] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260321 02:59:01.840185 12738 raft_consensus.cc:2749] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:01.819662 12787 consensus_queue.cc:1059] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260321 02:59:01.839356 12749 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:01.877084 12722 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:01.906525 12783 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:01.840469 12794 raft_consensus.cc:2804] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:01.908170 12794 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:01.903661 12730 raft_consensus.cc:2749] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:01.905079 12761 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:01.914929 12761 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:01.908571 12794 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:01.917215 12783 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:01.918589 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 81715.4us
W20260321 02:59:01.919287 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:01.877377 12795 raft_consensus.cc:3055] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 LEADER]: Stepping down as leader of term 3
W20260321 02:59:01.881763 12786 log.cc:927] Time spent T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.057s user 0.000s sys 0.002s
W20260321 02:59:01.885548 12788 log.cc:927] Time spent T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.066s user 0.001s sys 0.001s
W20260321 02:59:01.886168 12789 log.cc:927] Time spent T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
I20260321 02:59:01.887212 12434 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:01.873346 12536 raft_consensus.cc:2804] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election won for term 3
W20260321 02:59:01.873293 12790 log.cc:927] Time spent T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
I20260321 02:59:01.921007 12795 raft_consensus.cc:740] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
W20260321 02:59:01.948987 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 77459.8us
I20260321 02:59:01.917883 12787 raft_consensus.cc:3055] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:59:01.925205 12722 consensus_queue.cc:1048] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260321 02:59:01.958273 11383 raft_consensus.cc:1240] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 3. Current term is 4. Ops: []
W20260321 02:59:01.951833 11349 leader_election.cc:336] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.944s (SENT)
I20260321 02:59:01.921802 12679 raft_consensus.cc:3055] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
W20260321 02:59:01.954246 11942 leader_election.cc:336] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.751s (SENT)
I20260321 02:59:01.959841 11942 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:01.961126 11942 leader_election.cc:336] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Timed out: RequestConsensusVote RPC to 127.9.213.1:36717 timed out after 1.768s (SENT)
W20260321 02:59:01.962008 11942 leader_election.cc:336] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.768s (SENT)
I20260321 02:59:01.962320 11942 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:01.955673 12787 raft_consensus.cc:740] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:01.956226 11275 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:01.967553 11349 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:01.967343 11132 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42502 (request call id 222) took 1758 ms. Trace:
I20260321 02:59:01.968294 11132 rpcz_store.cc:276] 0321 02:59:00.208959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:00.260572 (+ 51613us) service_pool.cc:225] Handling call
0321 02:59:01.967282 (+1706710us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:59:01.557057 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:59:01.557953 (+ 896us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:59:01.558001 (+ 48us) write_op.cc:435] Acquired schema lock
0321 02:59:01.558027 (+ 26us) tablet.cc:661] Decoding operations
0321 02:59:01.559146 (+ 1119us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:59:01.559222 (+ 76us) write_op.cc:641] Partition lock acquired for write op
0321 02:59:01.559263 (+ 41us) tablet.cc:684] Acquiring locks for 8 operations
0321 02:59:01.626013 (+ 66750us) tablet.cc:700] Row locks acquired
0321 02:59:01.628763 (+ 2750us) write_op.cc:260] PREPARE: finished
0321 02:59:01.640804 (+ 12041us) write_op.cc:270] Start()
0321 02:59:01.650913 (+ 10109us) write_op.cc:276] Timestamp: P: 1774061941638424 usec, L: 0
0321 02:59:01.653919 (+ 3006us) op_driver.cc:348] REPLICATION: starting
0321 02:59:01.709467 (+ 55548us) log.cc:844] Serialized 3205 byte log entry
0321 02:59:01.734379 (+ 24912us) op_driver.cc:464] REPLICATION: finished
0321 02:59:01.735247 (+ 868us) write_op.cc:301] APPLY: starting
0321 02:59:01.735357 (+ 110us) tablet.cc:1366] starting BulkCheckPresence
0321 02:59:01.735634 (+ 277us) tablet.cc:1369] finished BulkCheckPresence
0321 02:59:01.735787 (+ 153us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:59:01.737529 (+ 1742us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:59:01.737726 (+ 197us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0321 02:59:01.737768 (+ 42us) write_op.cc:312] APPLY: finished
0321 02:59:01.738039 (+ 271us) log.cc:844] Serialized 65 byte log entry
0321 02:59:01.738280 (+ 241us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:01.738442 (+ 162us) write_op.cc:454] Released schema lock
0321 02:59:01.738524 (+ 82us) write_op.cc:341] FINISH: Updating metrics
0321 02:59:01.740769 (+ 2245us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:01.740802 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":246453,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":321,"apply.run_cpu_time_us":3665,"apply.run_wall_time_us":5499,"num_ops":8,"prepare.queue_time_us":640,"prepare.run_cpu_time_us":5756,"prepare.run_wall_time_us":173071,"replication_time_us":78737,"spinlock_wait_cycles":1772416}]]}
I20260321 02:59:01.972457 12761 raft_consensus.cc:2804] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:01.972916 12761 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:01.972950 12787 consensus_queue.cc:260] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:01.973846 12761 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:01.952314 12536 raft_consensus.cc:697] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:01.975705 12749 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:01.975848 12536 consensus_queue.cc:237] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:01.978317 12749 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:01.978983 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 160616us
W20260321 02:59:01.972065 11942 leader_election.cc:336] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Timed out: RequestConsensusVote RPC to 127.9.213.1:36717 timed out after 1.552s (SENT)
I20260321 02:59:01.967303 12701 raft_consensus.cc:2749] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260321 02:59:01.980428 11942 leader_election.cc:336] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.817s (SENT)
I20260321 02:59:01.981930 11942 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:01.982616 11200 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:01.967350 12797 raft_consensus.cc:2749] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:01.983412 12738 raft_consensus.cc:2749] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:01.966743 12679 raft_consensus.cc:740] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:01.983451 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:01.986613 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:01.987730 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:01.994765 11306 raft_consensus.cc:2393] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 2: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:01.992931 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:01.992942 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:01.987371 11200 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:01.996850 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:01.996923 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:01.997633 11302 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:01.995584 11308 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:02.006943 11200 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:02.006853 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:02.011787 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:02.013039 11200 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:02.013331 11304 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:02.011608 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:02.015370 11307 raft_consensus.cc:2393] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:02.018740 11200 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:01.983472 12753 raft_consensus.cc:2749] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260321 02:59:02.016968 11943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:02.027225 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:02.019129 12736 raft_consensus.cc:2749] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:01.982225 12722 consensus_queue.cc:1059] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260321 02:59:01.998852 12797 raft_consensus.cc:2749] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:02.049469 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:01.992254 12723 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:01.982498 12715 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:01.997056 12757 consensus_queue.cc:1048] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
W20260321 02:59:02.023351 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 71623.2us
I20260321 02:59:02.059449 11311 raft_consensus.cc:1240] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 2. Current term is 3. Ops: []
I20260321 02:59:02.021785 12738 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:02.019275 12799 raft_consensus.cc:2804] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260321 02:59:02.126609 12790 log.cc:927] Time spent T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
W20260321 02:59:02.129813 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 110121us
W20260321 02:59:02.146205 11349 leader_election.cc:336] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 2.055s (SENT)
I20260321 02:59:02.146147 12715 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:02.149130 12723 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:02.150153 11305 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 2.
I20260321 02:59:02.196518 12799 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:02.197573 11200 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:02.198746 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:02.198908 12799 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
W20260321 02:59:02.150597 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:02.199604 11305 raft_consensus.cc:2393] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 4: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:02.200430 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:02.200610 12757 raft_consensus.cc:2804] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:02.201557 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:02.202945 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:02.023519 12792 raft_consensus.cc:2804] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:02.204502 12792 raft_consensus.cc:697] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:02.205462 12792 consensus_queue.cc:237] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.206297 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:02.206601 11383 raft_consensus.cc:1240] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:02.021690 12720 consensus_queue.cc:1048] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260321 02:59:02.209084 12799 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.152100 12784 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.088454 11309 raft_consensus.cc:2368] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 2. Current term is 3.
I20260321 02:59:02.154182 12736 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:02.215968 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:02.215759 12736 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.217010 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:02.154662 12794 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.027454 12795 consensus_queue.cc:260] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:02.225616 12799 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:02.225808 12795 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:02.215332 12784 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:01.997705 12785 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.022162 12749 raft_consensus.cc:2749] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:02.162881 12738 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.062343 12679 consensus_queue.cc:260] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:02.167382 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:02.168581 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:02.166213 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:02.169706 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:02.170047 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:02.171049 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:02.171322 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:02.172358 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:02.247340 11303 raft_consensus.cc:2393] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 2: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
I20260321 02:59:02.173484 12783 raft_consensus.cc:3055] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:59:02.180045 12787 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:02.182090 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:02.184358 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:02.185397 11233 raft_consensus.cc:1240] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:02.183728 11230 raft_consensus.cc:1240] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:02.186342 11236 raft_consensus.cc:1240] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:02.188802 11235 raft_consensus.cc:1240] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:02.189059 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:02.190090 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:02.190610 11134 catalog_manager.cc:5671] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 3 to 4, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 4 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:02.191200 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:02.191897 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:02.193280 12722 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:02.193466 12761 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:02.063011 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 67567.3us
I20260321 02:59:02.207688 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:02.197011 11349 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:02.197582 11275 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:02.254127 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 56584.1us
I20260321 02:59:02.218068 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:02.254546 11134 catalog_manager.cc:5671] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 3 to 4, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 4 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:02.215380 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:02.217788 12736 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:02.225059 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:02.242283 12757 raft_consensus.cc:697] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:02.241326 11200 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:02.268052 11942 leader_election.cc:336] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.594s (SENT)
I20260321 02:59:02.241900 11379 raft_consensus.cc:1275] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:59:02.268548 11942 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:02.245347 11304 raft_consensus.cc:2410] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260321 02:59:02.245507 11307 raft_consensus.cc:2393] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 4: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:02.172663 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:02.256146 11229 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:02.256731 11231 raft_consensus.cc:2393] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 4: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:02.261816 11385 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:02.266613 12723 raft_consensus.cc:2804] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:02.270838 12723 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:02.271298 12723 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:02.274004 12761 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:02.275430 11378 raft_consensus.cc:1275] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:59:02.264827 11383 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:02.275779 11275 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:02.276944 12679 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:02.278254 12730 raft_consensus.cc:2749] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:02.278963 11275 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:02.279452 12730 consensus_queue.cc:1059] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:02.280413 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:02.264243 12722 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.281344 12738 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:02.282508 12496 raft_consensus.cc:2749] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:02.282866 12722 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:02.283497 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:02.284220 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 138035us
I20260321 02:59:02.284663 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:02.284823 11349 leader_election.cc:336] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 2.097s (SENT)
I20260321 02:59:02.285357 11200 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:02.285646 11349 leader_election.cc:336] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 2.102s (SENT)
I20260321 02:59:02.285732 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:02.286165 11349 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:02.287191 11308 raft_consensus.cc:1240] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260321 02:59:02.265684 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:02.234011 12784 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:02.300246 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:02.237427 11376 raft_consensus.cc:3055] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:02.287645 12753 raft_consensus.cc:2749] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:02.288570 12736 consensus_queue.cc:1048] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:02.322530 11376 raft_consensus.cc:740] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:02.290490 12793 consensus_queue.cc:1059] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:02.291417 12536 raft_consensus.cc:2804] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:59:02.325965 11383 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:59:02.326840 11200 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:02.327602 11385 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:02.328058 11200 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:02.258447 12783 raft_consensus.cc:740] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:02.294656 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:02.261124 11134 catalog_manager.cc:5671] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 1 to 2, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 2 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:02.296484 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:02.287874 11349 leader_election.cc:336] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 2.155s (SENT)
I20260321 02:59:02.299326 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:02.334478 11311 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:02.318303 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:02.335934 11309 raft_consensus.cc:2393] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 4: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:02.318878 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:02.319942 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:02.301851 11304 raft_consensus.cc:2368] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:59:02.301059 11306 raft_consensus.cc:2393] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
W20260321 02:59:02.321949 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 53916.5us
I20260321 02:59:02.318102 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:02.289127 12785 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:02.340271 11310 raft_consensus.cc:2393] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:02.322882 11308 raft_consensus.cc:1240] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260321 02:59:02.264623 11975 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:02.342346 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:02.346632 12802 consensus_queue.cc:1059] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260321 02:59:02.348428 11307 raft_consensus.cc:2393] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 4: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:02.350080 11307 raft_consensus.cc:1240] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:02.325538 12761 consensus_queue.cc:1059] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260321 02:59:02.342689 11942 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:02.352814 12804 raft_consensus.cc:2804] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:02.354446 11134 catalog_manager.cc:5671] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 1 to 2, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 2 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:02.342106 12797 consensus_queue.cc:1059] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260321 02:59:02.345068 11349 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:02.356415 12434 consensus_queue.cc:1059] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:59:02.342705 11310 raft_consensus.cc:1240] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:02.357419 11307 raft_consensus.cc:1240] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260321 02:59:02.358644 11308 raft_consensus.cc:1240] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260321 02:59:02.300254 12496 raft_consensus.cc:2749] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:02.364239 11229 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:02.324764 12801 raft_consensus.cc:2749] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:02.370141 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:02.372607 11308 raft_consensus.cc:1240] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260321 02:59:02.344202 11376 consensus_queue.cc:260] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.374532 11376 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:02.356088 12701 raft_consensus.cc:2749] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:02.332105 12794 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:02.382164 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 94302.8us
I20260321 02:59:02.391201 11311 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:02.393522 11200 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:02.363719 12720 raft_consensus.cc:2749] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:59:02.324420 12784 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.393728 11311 raft_consensus.cc:1240] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: []
W20260321 02:59:02.395148 11349 leader_election.cc:336] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 2.171s (SENT)
I20260321 02:59:02.425516 11309 raft_consensus.cc:1240] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:02.401839 11308 raft_consensus.cc:1240] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:02.429680 11308 raft_consensus.cc:1275] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:59:02.407056 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:02.373701 12804 raft_consensus.cc:697] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
W20260321 02:59:02.420305 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:02.434113 11303 raft_consensus.cc:1275] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260321 02:59:02.434487 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:02.400221 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:02.412882 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 91724.5us
I20260321 02:59:02.399386 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:02.432638 11308 raft_consensus.cc:3055] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Stepping down as leader of term 1
W20260321 02:59:02.436295 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 75584.6us
I20260321 02:59:02.437278 11275 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:02.438588 12810 consensus_queue.cc:1059] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:02.440335 12810 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:02.440912 12810 raft_consensus.cc:697] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 4 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:02.441922 12810 consensus_queue.cc:237] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.444298 12794 raft_consensus.cc:3055] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:59:02.444834 12794 raft_consensus.cc:740] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:02.445992 12794 consensus_queue.cc:260] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.447427 12794 raft_consensus.cc:3060] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:02.437294 11942 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:02.436529 12793 consensus_queue.cc:1059] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
W20260321 02:59:02.405573 11350 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 134821us
I20260321 02:59:02.363412 12792 consensus_queue.cc:1048] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260321 02:59:02.429976 11376 raft_consensus.cc:1275] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260321 02:59:02.365561 12752 consensus_queue.cc:1048] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20260321 02:59:02.435683 11308 raft_consensus.cc:740] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:02.457049 11942 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:02.449419 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 66423.9us
I20260321 02:59:02.408975 12496 raft_consensus.cc:2749] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:59:02.461525 11229 raft_consensus.cc:1240] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:02.395257 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:02.459667 11302 raft_consensus.cc:2368] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:59:02.463626 11231 raft_consensus.cc:1240] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:02.430748 12749 raft_consensus.cc:2764] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260321 02:59:02.471541 11942 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:02.464155 12797 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:02.406796 12736 consensus_queue.cc:1059] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20260321 02:59:02.473531 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:02.473774 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:02.469657 11308 consensus_queue.cc:260] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.475112 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:02.433157 12802 consensus_queue.cc:1059] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.042s
I20260321 02:59:02.473174 12655 raft_consensus.cc:2749] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:02.489130 11308 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:02.493286 12817 consensus_queue.cc:1059] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:02.493634 12736 raft_consensus.cc:3055] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:02.494107 12753 raft_consensus.cc:3055] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:02.494288 12736 raft_consensus.cc:740] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:02.494571 12753 raft_consensus.cc:740] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:02.496356 12749 raft_consensus.cc:3055] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:02.496907 12749 raft_consensus.cc:740] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:02.496973 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:02.497694 11302 raft_consensus.cc:3060] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:02.498145 12749 consensus_queue.cc:260] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.495538 12753 consensus_queue.cc:260] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.500303 12753 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:02.495561 12736 consensus_queue.cc:260] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.503456 12736 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:02.509479 12817 consensus_queue.cc:1048] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260321 02:59:02.510938 12808 log.cc:927] Time spent T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
I20260321 02:59:02.511421 11304 raft_consensus.cc:1240] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:02.513532 12434 raft_consensus.cc:3055] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:59:02.513947 12434 raft_consensus.cc:740] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:02.516726 11311 raft_consensus.cc:1240] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260321 02:59:02.518263 11311 raft_consensus.cc:1240] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:02.430651 12757 consensus_queue.cc:237] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.505141 12749 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:02.516494 12812 raft_consensus.cc:2749] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:02.504647 11302 raft_consensus.cc:2468] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 2.
I20260321 02:59:02.484309 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:02.544648 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:02.545929 11304 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:02.460134 12783 consensus_queue.cc:260] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.548462 12434 consensus_queue.cc:260] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:02.548604 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 86030.6us
I20260321 02:59:02.544406 11200 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:02.550444 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 72449.5us
W20260321 02:59:02.549580 11349 leader_election.cc:336] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 2.223s (SENT)
I20260321 02:59:02.520694 12722 raft_consensus.cc:3055] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:59:02.552065 11942 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:02.550699 11200 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:02.553592 11349 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:02.475437 12816 consensus_queue.cc:1059] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260321 02:59:02.556424 12434 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:02.569365 11942 leader_election.cc:400] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267) with higher term. Message: Invalid argument: T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 2. Current term is 3.
I20260321 02:59:02.574512 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:02.575526 11377 raft_consensus.cc:2410] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260321 02:59:02.576260 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:02.594235 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 44673.3us
I20260321 02:59:02.596307 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:02.613591 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:02.533614 12761 consensus_queue.cc:1059] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
W20260321 02:59:02.636193 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:02.640959 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:02.647645 11349 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:02.561599 12812 raft_consensus.cc:2804] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:59:02.555223 12819 raft_consensus.cc:2804] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:59:02.571710 12803 raft_consensus.cc:2749] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:59:02.557152 12536 raft_consensus.cc:697] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
W20260321 02:59:02.652195 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:02.552908 12752 consensus_queue.cc:1048] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.082s
I20260321 02:59:02.555701 12496 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:02.654901 11304 raft_consensus.cc:1240] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 2. Current term is 3. Ops: [2.3-2.3]
I20260321 02:59:02.565435 12809 raft_consensus.cc:2749] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:02.586195 12715 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:02.656497 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:02.675434 11236 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:02.556187 12722 raft_consensus.cc:740] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
W20260321 02:59:02.659251 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:02.641731 12793 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:02.672484 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:02.669270 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:02.682523 11304 raft_consensus.cc:2410] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260321 02:59:02.658845 12783 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:02.663316 12784 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:02.640962 12792 consensus_queue.cc:1048] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
W20260321 02:59:02.696945 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 101248us
I20260321 02:59:02.681807 12730 raft_consensus.cc:3055] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:02.700125 12730 raft_consensus.cc:740] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:02.701241 12730 consensus_queue.cc:260] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.702628 12730 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:02.688042 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:02.695554 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:02.707664 11302 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:02.709156 11304 raft_consensus.cc:1275] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260321 02:59:02.711920 11304 raft_consensus.cc:1240] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:02.675287 12812 raft_consensus.cc:697] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
W20260321 02:59:02.777454 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:02.855014 12806 log.cc:927] Time spent T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260321 02:59:02.677773 12496 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:02.907405 12801 raft_consensus.cc:3055] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:59:02.685655 12715 raft_consensus.cc:697] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
W20260321 02:59:02.905172 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 257762us
I20260321 02:59:02.718418 12723 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.693583 12793 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.552759 12804 consensus_queue.cc:237] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:02.901435 12795 consensus_meta.cc:287] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 0.675s user 0.010s sys 0.001s
I20260321 02:59:02.856125 12809 raft_consensus.cc:2749] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:02.910074 11349 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:02.906970 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:02.900532 12787 consensus_meta.cc:287] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.649s user 0.011s sys 0.000s
I20260321 02:59:03.005936 11306 raft_consensus.cc:1240] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:03.008064 11229 raft_consensus.cc:1275] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260321 02:59:03.008474 11229 pending_rounds.cc:85] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb: Aborting all ops after (but not including) 1
I20260321 02:59:03.008749 11229 pending_rounds.cc:107] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260321 02:59:03.009047 11229 raft_consensus.cc:2889] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:03.009526 11302 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
W20260321 02:59:03.009912 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 99915.3us
I20260321 02:59:03.011045 11349 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:02.990180 12785 raft_consensus.cc:3055] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:02.908648 12715 consensus_queue.cc:237] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.907846 12801 raft_consensus.cc:740] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:03.025820 12434 raft_consensus.cc:3055] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:59:03.030346 12434 raft_consensus.cc:740] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
W20260321 02:59:03.010599 12824 log.cc:927] Time spent T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.456s user 0.002s sys 0.000s
I20260321 02:59:03.039892 12434 consensus_queue.cc:260] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:02.899571 12722 consensus_queue.cc:260] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:03.013330 12830 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:02.905318 12536 consensus_queue.cc:237] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:03.018991 12720 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:03.020169 12785 raft_consensus.cc:740] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:03.035481 12811 raft_consensus.cc:3055] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Stepping down as leader of term 1
W20260321 02:59:03.065596 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:03.061852 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:03.062616 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:03.049494 11227 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
W20260321 02:59:03.087209 11310 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.013s sys 0.013s
I20260321 02:59:03.051738 12809 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
W20260321 02:59:03.090842 11310 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s user 0.014s sys 0.013s
W20260321 02:59:03.092451 11275 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.005s sys 0.029s
W20260321 02:59:03.093125 11275 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.613s user 0.005s sys 0.029s
W20260321 02:59:03.093951 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 614053us
I20260321 02:59:03.097432 11228 raft_consensus.cc:1240] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
W20260321 02:59:03.098274 11275 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:03.098670 11275 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:03.098958 11275 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:03.099396 11275 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:03.103315 11308 consensus_meta.cc:287] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.613s user 0.007s sys 0.000s
I20260321 02:59:03.063043 12811 raft_consensus.cc:740] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:03.103901 11308 raft_consensus.cc:1275] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260321 02:59:03.090996 12822 log.cc:927] Time spent T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.218s user 0.001s sys 0.000s
W20260321 02:59:03.094201 12797 consensus_meta.cc:287] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 0.617s user 0.010s sys 0.000s
I20260321 02:59:03.120028 12797 raft_consensus.cc:2749] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260321 02:59:03.120905 11975 consensus_meta.cc:287] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 0.774s user 0.011s sys 0.002s
I20260321 02:59:03.121308 11975 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:03.122151 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:03.123737 11276 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:03.058525 12830 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:03.131184 12783 raft_consensus.cc:2804] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:59:03.132135 12783 raft_consensus.cc:697] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:03.133139 12783 consensus_queue.cc:237] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:03.134609 12828 log.cc:927] Time spent T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.142s user 0.000s sys 0.000s
I20260321 02:59:03.048966 12812 consensus_queue.cc:237] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:03.136560 12753 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:03.137145 12753 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:03.090066 12496 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:03.142258 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:03.062063 12720 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:03.088135 12842 raft_consensus.cc:2749] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260321 02:59:03.147435 11311 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.008s sys 0.022s
W20260321 02:59:03.148681 11311 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s user 0.008s sys 0.022s
I20260321 02:59:03.139386 12753 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:03.167189 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:03.113101 12838 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:03.142578 12819 raft_consensus.cc:697] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:03.108114 12809 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:03.197228 12815 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:03.197964 12815 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:03.199150 12809 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:03.143261 12801 consensus_queue.cc:260] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:03.201198 12801 raft_consensus.cc:3060] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:03.204268 12834 raft_consensus.cc:2749] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:03.195113 12811 consensus_queue.cc:260] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:03.206888 12811 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
W20260321 02:59:03.207000 12736 consensus_meta.cc:287] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.701s user 0.010s sys 0.000s
I20260321 02:59:03.193311 12723 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:03.191022 12838 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:03.141367 12785 consensus_queue.cc:260] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:03.219048 12785 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:03.184826 11227 raft_consensus.cc:606] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:03.223794 11227 raft_consensus.cc:614] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260321 02:59:03.224601 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:03.190641 12793 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:03.229400 12679 consensus_meta.cc:287] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.952s user 0.012s sys 0.000s
I20260321 02:59:03.269975 11942 leader_election.cc:304] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:03.276593 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:03.291463 12655 raft_consensus.cc:2804] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader election won for term 2
W20260321 02:59:03.292557 12794 consensus_meta.cc:287] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.845s user 0.014s sys 0.000s
I20260321 02:59:03.299939 12655 raft_consensus.cc:697] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:03.300627 12815 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:03.301085 12655 consensus_queue.cc:237] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:03.312361 12838 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:03.313794 12720 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:03.318203 12819 consensus_queue.cc:237] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:03.319730 12738 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:03.320459 11307 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1263) took 1019 ms. Trace:
I20260321 02:59:03.320529 12738 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:03.320992 11307 rpcz_store.cc:276] 0321 02:59:02.300671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:02.449883 (+149212us) service_pool.cc:225] Handling call
0321 02:59:02.498769 (+ 48886us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:02.532224 (+ 33455us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:02.532255 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:02.907082 (+374827us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b2c00010da8. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8aae48c81 00007fa8aae585e2 00007fa8ac587de1 00007fa8ac54c81a 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22
0321 02:59:02.907533 (+ 451us) log.cc:844] Serialized 26 byte log entry
0321 02:59:03.081811 (+174278us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:59:03.081848 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:03.081916 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:03.227868 (+145952us) raft_consensus.cc:1685] finished
0321 02:59:03.227902 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:03.320263 (+ 92361us) spinlock_profiling.cc:243] Waited 27.7 ms on lock 0x7b54001969d0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54bc8d 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:03.320401 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":40659,"raft.queue_time_us":38075,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":11268,"spinlock_wait_cycles":68050688,"thread_start_us":173889,"threads_started":1,"wal-append.queue_time_us":283398}
I20260321 02:59:03.323912 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:03.324374 12434 raft_consensus.cc:3060] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:03.327566 11376 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "513757b127054285b0686cd45044c596"
dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:44010
I20260321 02:59:03.328387 11376 raft_consensus.cc:606] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:03.290853 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:03.329535 11975 raft_consensus.cc:2368] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:59:03.324110 12738 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:03.330663 12722 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:03.337280 11227 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:03.339073 12830 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:03.291091 12834 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:03.339216 12749 consensus_meta.cc:287] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.794s user 0.006s sys 0.006s
I20260321 02:59:03.324713 11973 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
I20260321 02:59:03.365788 11231 raft_consensus.cc:1240] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260321 02:59:03.328770 12730 consensus_meta.cc:287] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.626s user 0.013s sys 0.002s
I20260321 02:59:03.373921 11276 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:03.375746 11350 leader_election.cc:400] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285) with higher term. Message: Invalid argument: T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:59:03.376230 11350 leader_election.cc:403] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260321 02:59:03.340281 12834 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:03.382300 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 371286us
I20260321 02:59:03.366607 12810 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:03.401151 11235 raft_consensus.cc:1240] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:03.402069 11349 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:03.405405 11349 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:03.383752 12730 raft_consensus.cc:2804] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:03.382980 12804 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:03.408401 11303 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.877s user 0.004s sys 0.009s
W20260321 02:59:03.410558 11303 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.879s user 0.005s sys 0.009s
W20260321 02:59:03.411155 11383 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.714s user 0.000s sys 0.014s
W20260321 02:59:03.411844 11383 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.714s user 0.000s sys 0.014s
I20260321 02:59:03.411883 11349 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:03.414366 11384 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.758s user 0.008s sys 0.006s
I20260321 02:59:03.385788 12849 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 3
I20260321 02:59:03.406105 12830 raft_consensus.cc:2749] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260321 02:59:03.414685 11384 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.759s user 0.008s sys 0.006s
I20260321 02:59:03.406694 12816 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:03.410414 12730 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:03.422227 11201 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.000s sys 0.016s
I20260321 02:59:03.410053 12809 raft_consensus.cc:2749] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:03.420545 11349 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:03.423425 11302 raft_consensus.cc:1240] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 2. Current term is 3. Ops: []
W20260321 02:59:03.424031 11201 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.653s user 0.001s sys 0.016s
I20260321 02:59:03.431633 11349 leader_election.cc:304] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
W20260321 02:59:03.434036 11942 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:03.406937 12810 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:03.435249 11200 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.859s user 0.005s sys 0.019s
W20260321 02:59:03.435670 11200 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.860s user 0.005s sys 0.019s
I20260321 02:59:03.435539 11310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56616 (request call id 1416) took 1257 ms. Trace:
I20260321 02:59:03.436198 11310 rpcz_store.cc:276] 0321 02:59:02.178474 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:02.366165 (+187691us) service_pool.cc:225] Handling call
0321 02:59:02.366719 (+ 554us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:02.377861 (+ 11142us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:02.377912 (+ 51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:02.378501 (+ 589us) log.cc:844] Serialized 26 byte log entry
0321 02:59:03.093022 (+714521us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:03.149540 (+ 56518us) log.cc:844] Serialized 14 byte log entry
0321 02:59:03.318420 (+168880us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b18003a73e0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b2300441 00007fa8b230cca1 00007fa8a942a63a 00007fa8b2aa65c9 00007fa8a92e8c50 00007fa8a94b32b4 00007fa8a94b3225 00007fa8ac544440 00007fa8ac556af6 00007fa8ac567667 00007fa8ac5675e8
0321 02:59:03.375728 (+ 57308us) spinlock_profiling.cc:243] Waited 1.93 ms on lock 0x7b18003a73e0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b2300441 00007fa8b230cca1 00007fa8a94b61c3 00007fa8a94b8591 00007fa8a94b8549 00007fa8a94b84d9 00007fa8a94b84a1 00007fa8a94b779d 0000000000379254 0000000000379099 00007fa8a949974a
0321 02:59:03.409641 (+ 33913us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:03.411527 (+ 1886us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:03.435302 (+ 23775us) raft_consensus.cc:1685] finished
0321 02:59:03.435326 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:03.435478 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":26307,"prepare.queue_time_us":54275,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":208468,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":172,"spinlock_wait_cycles":32510720,"thread_start_us":751723,"threads_started":2,"wal-append.queue_time_us":738584}
W20260321 02:59:03.439895 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:03.440970 11349 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:03.444288 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 340876us
I20260321 02:59:03.425016 12730 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:03.413534 12804 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:03.446532 11200 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:03.443316 12757 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:03.449156 12848 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:03.449638 12715 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:03.435436 12738 raft_consensus.cc:2749] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:03.447839 11349 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:03.463696 11200 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:03.467638 11200 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:03.439836 12805 raft_consensus.cc:493] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:03.474733 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:03.443944 11227 raft_consensus.cc:606] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:03.489200 11227 raft_consensus.cc:614] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:03.439426 12859 raft_consensus.cc:2804] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:59:03.444593 12809 raft_consensus.cc:2749] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:03.470293 12803 raft_consensus.cc:2749] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260321 02:59:03.492830 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 45010.4us
I20260321 02:59:03.490566 12785 consensus_queue.cc:1059] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260321 02:59:03.485874 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:03.483932 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:03.485131 12783 raft_consensus.cc:476] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 2 LEADER]: Not starting pre-election -- already a leader
I20260321 02:59:03.488713 12842 raft_consensus.cc:2749] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:03.497682 11231 raft_consensus.cc:2393] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 5: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:03.424736 12816 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:03.491590 11303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1262) took 1226 ms. Trace:
I20260321 02:59:03.466959 12757 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:03.483911 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:03.457194 12830 raft_consensus.cc:2749] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:03.505558 12810 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:03.506686 12834 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:03.508177 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:03.508903 11234 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:03.509474 12849 raft_consensus.cc:2749] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285) with higher term. Message: Invalid argument: T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:59:03.502132 12720 consensus_queue.cc:1048] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.009s
I20260321 02:59:03.455654 12715 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:03.512174 11234 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58214 (request call id 992) took 1057 ms. Trace:
I20260321 02:59:03.498970 11200 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:03.499142 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:03.497144 11230 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
I20260321 02:59:03.500403 11228 raft_consensus.cc:1240] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:03.481216 12536 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:03.501255 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:03.481050 12799 raft_consensus.cc:2749] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:03.497905 12749 consensus_queue.cc:1048] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
W20260321 02:59:03.512715 11944 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.857s user 0.013s sys 0.013s
I20260321 02:59:03.495186 11349 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:03.501736 11303 rpcz_store.cc:276] 0321 02:59:02.267808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:02.434817 (+167009us) service_pool.cc:225] Handling call
0321 02:59:02.468006 (+ 33189us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:02.474291 (+ 6285us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:02.474320 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:02.477828 (+ 3508us) log.cc:844] Serialized 26 byte log entry
0321 02:59:03.412778 (+934950us) raft_consensus.cc:1645] Marking committed up to 5
0321 02:59:03.420680 (+ 7902us) log.cc:844] Serialized 14 byte log entry
0321 02:59:03.421060 (+ 380us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:03.421163 (+ 103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:03.491269 (+ 70106us) raft_consensus.cc:1685] finished
0321 02:59:03.491291 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:03.491429 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15895,"prepare.queue_time_us":7577,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":39699,"raft.run_cpu_time_us":382,"raft.run_wall_time_us":8922,"thread_start_us":881439,"threads_started":1,"wal-append.queue_time_us":1002480}
W20260321 02:59:03.516896 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:03.518539 11944 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.914s user 0.014s sys 0.013s
I20260321 02:59:03.461000 12848 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:03.471416 12814 consensus_queue.cc:1048] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260321 02:59:03.516460 12720 raft_consensus.cc:2749] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:03.514134 11234 rpcz_store.cc:276] 0321 02:59:02.454502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:02.463938 (+ 9436us) service_pool.cc:225] Handling call
0321 02:59:03.511934 (+1047996us) spinlock_profiling.cc:243] Waited 795 ms on lock 0x7b540037ad54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:03.512088 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1828412160}
I20260321 02:59:03.528555 12816 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:03.522778 11940 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:03.502477 12803 raft_consensus.cc:3055] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Stepping down as leader of term 4
W20260321 02:59:03.528966 12860 log.cc:927] Time spent T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.106s user 0.000s sys 0.001s
I20260321 02:59:03.538267 12803 raft_consensus.cc:740] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:03.530593 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:03.523613 12722 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:03.502465 12859 raft_consensus.cc:697] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:03.530673 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:03.544361 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:03.544122 12722 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:03.545059 11972 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:03.496284 12854 raft_consensus.cc:2749] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:03.519351 12536 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:03.495335 12805 raft_consensus.cc:515] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:03.523864 12856 consensus_queue.cc:1059] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260321 02:59:03.552515 12856 raft_consensus.cc:3055] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:03.532200 12749 consensus_queue.cc:1059] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:59:03.552981 12536 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:03.533147 12715 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:03.534243 12720 consensus_queue.cc:1048] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:59:03.552997 12856 raft_consensus.cc:740] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:03.554195 12819 raft_consensus.cc:3055] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:59:03.535135 12757 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:03.535544 12799 consensus_queue.cc:1059] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:03.554754 12819 raft_consensus.cc:740] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:03.536954 11200 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:03.526906 12842 raft_consensus.cc:3060] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:03.537047 11228 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:03.530926 11236 raft_consensus.cc:1275] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:59:03.540446 11275 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:03.526021 11349 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:03.539242 12803 consensus_queue.cc:260] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:03.531255 11133 catalog_manager.cc:5671] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 1 to 2, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4). New cstate: current_term: 2 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:59:03.544736 12859 consensus_queue.cc:237] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:03.529211 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:03.543838 11975 raft_consensus.cc:2468] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:03.531390 12838 consensus_queue.cc:1059] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260321 02:59:03.547915 12722 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:03.531534 12785 consensus_queue.cc:1059] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260321 02:59:03.568620 12536 consensus_queue.cc:1048] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:59:03.563844 12819 consensus_queue.cc:260] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:03.549860 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:03.569991 11385 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:03.570240 12819 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
W20260321 02:59:03.536139 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:03.570749 12803 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:03.557181 11228 raft_consensus.cc:606] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:03.571322 11228 raft_consensus.cc:614] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:03.572824 11229 raft_consensus.cc:1240] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 2. Current term is 3. Ops: []
I20260321 02:59:03.573426 11942 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:03.555323 12799 raft_consensus.cc:3055] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:03.553043 12805 leader_election.cc:290] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:03.560777 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:03.566955 12834 raft_consensus.cc:2749] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:03.576195 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:03.577401 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:03.562841 11349 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:03.565424 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:03.566020 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:03.579633 12701 raft_consensus.cc:2804] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:03.585716 11229 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:03.585548 11228 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:03.586926 11228 raft_consensus.cc:606] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:03.587276 11228 raft_consensus.cc:614] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:03.585548 11235 raft_consensus.cc:2376] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 2. Re-sending same reply.
I20260321 02:59:03.561394 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:03.561769 12720 consensus_queue.cc:1048] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260321 02:59:03.587157 12701 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:03.554908 12856 consensus_queue.cc:260] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:03.530305 12730 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:03.562358 12757 raft_consensus.cc:2804] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:03.537060 12804 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:03.560699 11201 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:03.575445 12799 raft_consensus.cc:740] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:03.517997 12849 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:03.530603 12848 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:03.596318 12701 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:03.588405 11349 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
W20260321 02:59:03.590411 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:03.575588 12749 raft_consensus.cc:3055] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:03.592289 11302 raft_consensus.cc:1275] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260321 02:59:03.599261 11311 raft_consensus.cc:1275] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260321 02:59:03.599560 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:03.602077 11305 raft_consensus.cc:1275] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260321 02:59:03.595453 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:03.582746 12852 raft_consensus.cc:2804] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:03.605931 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:03.588783 12834 raft_consensus.cc:2804] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:03.610385 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:03.612480 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:03.616766 11304 raft_consensus.cc:2468] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:59:03.620332 11228 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
W20260321 02:59:03.625200 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 64584.7us
I20260321 02:59:03.633625 11379 raft_consensus.cc:1240] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260321 02:59:03.636425 11200 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:03.640205 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:03.606077 12799 consensus_queue.cc:260] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:03.658890 11201 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:03.660271 11235 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:03.662392 11200 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:03.664076 11200 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:03.614406 12757 raft_consensus.cc:697] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:03.614957 12749 raft_consensus.cc:740] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:03.666879 12805 raft_consensus.cc:2804] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:03.673569 12805 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:03.674124 12805 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:03.625025 12809 consensus_queue.cc:1059] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.009s
I20260321 02:59:03.619115 12834 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:03.626487 12839 raft_consensus.cc:2749] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:03.683334 12784 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:03.633030 12730 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:03.632035 12852 raft_consensus.cc:697] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 4 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:03.659922 12856 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:03.691990 11377 raft_consensus.cc:3055] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:03.713392 11377 raft_consensus.cc:740] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:03.619855 12738 consensus_queue.cc:1059] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.062s
I20260321 02:59:03.667531 12797 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:03.714883 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 91149.7us
I20260321 02:59:03.716251 11235 raft_consensus.cc:606] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:03.716364 11275 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.9.213.3:43533 timed out after 1.521s (SENT) [suppressed 63 similar messages]
I20260321 02:59:03.716816 11235 raft_consensus.cc:614] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:03.718642 11349 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:03.722364 12835 consensus_queue.cc:1048] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:03.685290 12848 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:03.685750 12829 raft_consensus.cc:2749] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:03.693425 12720 raft_consensus.cc:2804] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:03.689917 12834 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
W20260321 02:59:03.728479 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:03.693245 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:03.700387 12804 raft_consensus.cc:2749] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:03.725706 11349 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:03.714321 12799 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:03.734830 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:03.749119 11972 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:03.752736 11381 raft_consensus.cc:2468] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
W20260321 02:59:03.770408 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:03.734515 12738 raft_consensus.cc:3055] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:03.712649 12784 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:03.733796 12838 raft_consensus.cc:3055] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:59:03.734055 12809 raft_consensus.cc:2749] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:03.737607 12720 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:03.788482 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 72123.8us
I20260321 02:59:03.789880 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:03.727129 12797 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:03.732295 12496 consensus_queue.cc:1059] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260321 02:59:03.800463 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260321 02:59:03.626995 12849 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:03.758126 12785 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:03.733862 12846 consensus_queue.cc:1048] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.062s
I20260321 02:59:03.802381 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:59:03.749889 12835 consensus_queue.cc:1048] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.041s
W20260321 02:59:03.794021 11275 leader_election.cc:336] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Timed out: RequestConsensusVote RPC to 127.9.213.3:43533 timed out after 1.551s (SENT)
W20260321 02:59:03.795804 12873 log.cc:927] Time spent T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.157s user 0.001s sys 0.000s
I20260321 02:59:03.780864 12738 raft_consensus.cc:740] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:03.807597 11275 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:03.809871 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:03.810607 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:03.815732 12877 raft_consensus.cc:3055] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:03.816318 12852 consensus_queue.cc:237] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:03.735184 12848 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:03.877393 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:03.829973 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 104650us
I20260321 02:59:03.796335 12720 raft_consensus.cc:3060] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:03.828835 11228 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:03.786978 12838 raft_consensus.cc:740] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:03.804630 12809 consensus_queue.cc:1059] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20260321 02:59:03.801975 12830 raft_consensus.cc:2804] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:03.861392 11385 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:03.863797 12738 consensus_queue.cc:260] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:03.871454 12701 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:03.865255 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 71244.3us
I20260321 02:59:03.867053 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:03.867990 12874 log.cc:927] Time spent T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.129s user 0.000s sys 0.000s
I20260321 02:59:03.868137 12829 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:03.869324 11973 raft_consensus.cc:1275] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260321 02:59:03.868566 11349 connection.cc:232] RPC call timeout handler was delayed by -1.5406736240000782s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16970000000000002s for a response
I20260321 02:59:03.873260 12797 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:03.899138 12830 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:03.860760 12811 raft_consensus.cc:2749] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:03.876696 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:03.738708 12757 consensus_queue.cc:237] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:03.892758 11200 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:03.862928 12877 raft_consensus.cc:740] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:03.864189 12749 consensus_queue.cc:260] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:03.899873 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:03.902070 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:04.003161 12877 consensus_queue.cc:260] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:04.004577 12877 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:04.004765 12876 log.cc:927] Time spent T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.176s user 0.000s sys 0.000s
I20260321 02:59:03.807140 12785 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:03.971520 11275 leader_election.cc:336] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Timed out: RequestConsensusVote RPC to 127.9.213.3:43533 timed out after 1.593s (SENT)
I20260321 02:59:04.009111 11275 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:03.895792 12814 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:03.980087 11385 raft_consensus.cc:1275] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260321 02:59:03.897270 12536 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:03.873631 11377 consensus_queue.cc:260] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:03.996924 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 104808us
I20260321 02:59:04.015251 11377 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:03.991482 12835 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:04.003223 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:04.016633 11200 leader_election.cc:304] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:03.993276 12830 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
W20260321 02:59:03.974114 11349 leader_election.cc:336] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.562s (SENT)
I20260321 02:59:04.021489 11349 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:04.010268 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:03.981505 12810 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:04.000397 12803 raft_consensus.cc:2749] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:04.009148 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:04.026857 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:04.019881 11307 raft_consensus.cc:1275] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260321 02:59:04.019742 11308 raft_consensus.cc:1275] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260321 02:59:04.003249 12848 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:04.013744 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:04.024435 11381 raft_consensus.cc:2410] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260321 02:59:04.025367 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:04.019742 12814 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:04.026150 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 52091.4us
I20260321 02:59:04.058951 11377 raft_consensus.cc:1275] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260321 02:59:04.080422 11349 connection.cc:232] RPC call timeout handler was delayed by -1.2950884560000304s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1855s for a response
W20260321 02:59:04.080921 11349 connection.cc:232] RPC call timeout handler was delayed by -1.249973624000063s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.202s for a response
W20260321 02:59:04.081213 11349 connection.cc:232] RPC call timeout handler was delayed by -1.1996884559999899s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19610000000000002s for a response
I20260321 02:59:04.081292 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:04.081527 11349 connection.cc:232] RPC call timeout handler was delayed by -1.025544468000021s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2167s for a response
I20260321 02:59:04.082239 11349 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:04.083366 11349 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:04.029655 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:04.030385 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:04.084264 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:04.084437 11376 raft_consensus.cc:2393] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 4: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:04.006836 12738 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:04.031837 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:03.972790 12829 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:04.085505 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 48856.8us
I20260321 02:59:04.086941 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:04.088208 11376 raft_consensus.cc:2393] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 4: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:04.088855 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:04.089334 11376 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:04.097160 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:04.098639 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:04.023217 12811 raft_consensus.cc:2749] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:04.097942 11382 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
W20260321 02:59:04.081424 12879 log.cc:927] Time spent T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
W20260321 02:59:04.104945 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:04.105742 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:04.031708 12810 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:04.107895 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:04.109053 11377 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:04.024225 12804 raft_consensus.cc:2804] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260321 02:59:04.118458 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:04.011714 12783 consensus_queue.cc:1048] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.082s
I20260321 02:59:04.024052 12722 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:04.134922 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:04.133834 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:04.218176 11275 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:04.218289 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:04.241616 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:04.101415 12839 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
W20260321 02:59:04.250038 11275 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:04.133708 12819 consensus_meta.cc:287] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.553s user 0.012s sys 0.001s
I20260321 02:59:04.108363 12856 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:04.251873 11275 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:04.219767 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 181974us
I20260321 02:59:04.261339 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:04.023491 12536 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:04.253083 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:04.224668 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:04.256043 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:04.097738 11976 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:04.023409 12835 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:04.258914 11382 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
W20260321 02:59:04.268052 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:04.033715 12805 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:04.020362 12838 consensus_queue.cc:260] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:04.272615 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:04.282228 11376 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:04.224951 12784 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:04.250520 12722 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:04.284020 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:04.274535 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:04.137108 12804 raft_consensus.cc:493] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:04.252882 12715 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:04.290277 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 206957us
I20260321 02:59:04.263465 12859 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:04.264073 12811 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:04.297731 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:04.290422 12898 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:04.299551 12898 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:04.301676 12898 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:04.281786 12885 raft_consensus.cc:2749] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:04.288079 11381 raft_consensus.cc:2376] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3. Re-sending same reply.
I20260321 02:59:04.288625 11382 raft_consensus.cc:2410] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260321 02:59:04.293562 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:04.275499 12849 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:04.283702 12496 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:04.312758 12829 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:04.313126 12496 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:04.315315 12496 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:04.317341 12536 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:04.317936 12536 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:04.291370 11349 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:04.318096 12810 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:04.285770 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:04.323539 11376 raft_consensus.cc:2410] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260321 02:59:04.267998 12856 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:04.312485 12814 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:04.318419 12536 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:04.353861 12536 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:04.290392 12804 raft_consensus.cc:3060] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:04.334223 11349 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:04.354983 12720 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:04.356467 11349 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:04.358273 11349 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:04.302772 12701 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:04.319742 12883 log.cc:927] Time spent T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.420s user 0.001s sys 0.000s
I20260321 02:59:04.337134 11378 raft_consensus.cc:2393] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 4: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:04.340497 12885 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:04.341274 12805 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:04.344087 12849 raft_consensus.cc:2749] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:04.329388 12859 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:04.345100 12893 log.cc:927] Time spent T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
W20260321 02:59:04.346256 12891 log.cc:927] Time spent T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.207s user 0.000s sys 0.001s
W20260321 02:59:04.348764 11972 consensus_meta.cc:287] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 0.562s user 0.010s sys 0.001s
I20260321 02:59:04.390419 11972 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:04.392311 12900 raft_consensus.cc:2749] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:04.394784 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:04.395110 12830 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:04.395741 11975 raft_consensus.cc:2368] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5.
I20260321 02:59:04.397742 12859 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:04.398435 11377 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:04.399501 12834 consensus_meta.cc:287] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.637s user 0.005s sys 0.004s
I20260321 02:59:04.350206 12835 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:04.426205 11385 raft_consensus.cc:1275] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:59:04.335096 11382 raft_consensus.cc:1240] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:04.428073 11382 raft_consensus.cc:1240] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260321 02:59:04.429607 11382 raft_consensus.cc:1240] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 LEADER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:04.303046 12896 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:04.317394 12722 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:04.322108 12757 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:04.324257 12897 raft_consensus.cc:2749] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260321 02:59:04.384393 11275 leader_election.cc:336] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Timed out: RequestConsensusVote RPC to 127.9.213.3:43533 timed out after 1.661s (SENT)
I20260321 02:59:04.440711 11275 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:04.442059 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 57694.2us
I20260321 02:59:04.302917 12715 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:04.446269 12715 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:04.447312 12783 raft_consensus.cc:2749] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:04.358899 12785 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:04.372154 12842 consensus_meta.cc:287] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.810s user 0.011s sys 0.001s
I20260321 02:59:04.386127 12536 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:04.407615 11377 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:04.400686 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:04.415601 12899 raft_consensus.cc:2749] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260321 02:59:04.418455 12902 log.cc:927] Time spent T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.090s user 0.001s sys 0.000s
I20260321 02:59:04.418967 12904 raft_consensus.cc:2749] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:04.420658 12901 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:04.420872 11201 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:04.422181 12889 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:04.422407 12856 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:04.424741 12894 log.cc:927] Time spent T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.105s user 0.000s sys 0.001s
I20260321 02:59:04.425376 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:04.384651 12885 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:04.510540 11276 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:04.359871 11349 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:04.323148 12811 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:04.435436 11382 raft_consensus.cc:1275] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260321 02:59:04.436244 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:04.436336 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:04.443634 11275 leader_election.cc:336] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Timed out: RequestConsensusVote RPC to 127.9.213.3:43533 timed out after 1.731s (SENT)
W20260321 02:59:04.494089 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 73336.3us
I20260321 02:59:04.495409 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:04.495848 11200 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:04.399919 12834 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:04.500177 11308 raft_consensus.cc:1240] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260321 02:59:04.448475 12907 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
W20260321 02:59:04.523577 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:04.505290 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:04.524415 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:04.524905 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:04.525134 11200 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:04.530056 12830 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:04.530737 11380 raft_consensus.cc:1275] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260321 02:59:04.464380 12536 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:04.532228 11376 raft_consensus.cc:1240] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:04.532461 12896 raft_consensus.cc:2749] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:04.534126 11201 leader_election.cc:400] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Vote denied by peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285) with higher term. Message: Invalid argument: T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5.
W20260321 02:59:04.505631 11384 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.008s sys 0.017s
W20260321 02:59:04.507840 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:04.454267 12757 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:04.515336 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 155626us
I20260321 02:59:04.516808 11275 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:04.516983 11383 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:04.540293 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:04.518191 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:04.551294 11942 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:04.518326 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:04.523108 11980 raft_consensus.cc:1275] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260321 02:59:04.497715 12785 raft_consensus.cc:2804] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:04.534590 11201 leader_election.cc:403] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260321 02:59:04.455366 12842 raft_consensus.cc:2749] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:04.517627 11972 raft_consensus.cc:2468] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
W20260321 02:59:04.570926 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 51418us
I20260321 02:59:04.527490 12903 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:04.509413 12904 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:04.559094 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 115489us
W20260321 02:59:04.577699 11275 leader_election.cc:336] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533): Timed out: RequestConsensusVote RPC to 127.9.213.3:43533 timed out after 1.734s (SENT)
I20260321 02:59:04.578267 11275 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:04.578508 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:04.560827 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:04.567155 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:04.570310 11349 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:04.580601 11383 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
W20260321 02:59:04.538843 11384 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.565s user 0.008s sys 0.017s
I20260321 02:59:04.584815 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:04.504731 12889 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:04.572170 12810 consensus_queue.cc:1048] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20260321 02:59:04.588205 11385 raft_consensus.cc:1240] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:04.589181 12899 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:04.575070 11378 raft_consensus.cc:1240] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 2. Current term is 3. Ops: []
I20260321 02:59:04.549846 12814 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:04.586221 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:04.550051 12783 raft_consensus.cc:2804] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:59:04.593827 11201 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:04.564329 12797 raft_consensus.cc:2804] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:04.564733 12898 raft_consensus.cc:2749] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:04.520978 12701 consensus_queue.cc:1048] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
W20260321 02:59:04.600133 11350 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 227323us
I20260321 02:59:04.554409 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:04.594367 11384 raft_consensus.cc:2393] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 3: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:04.609983 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:04.562216 12785 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:04.505162 12901 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:04.587110 12913 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:04.599032 12852 raft_consensus.cc:2749] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:04.597095 12897 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:04.533542 12907 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:04.615936 12835 raft_consensus.cc:2804] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:04.613912 12797 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:04.626250 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 51725.1us
I20260321 02:59:04.633673 12785 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:04.607889 12842 consensus_queue.cc:1059] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.044s
W20260321 02:59:04.640074 12809 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.022s sys 0.030s
I20260321 02:59:04.598217 12903 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:04.607028 12904 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:04.661764 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:04.674587 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:04.608309 12814 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:04.635665 12819 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
W20260321 02:59:04.667518 12809 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.701s user 0.022s sys 0.030s
I20260321 02:59:04.671435 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:04.671670 12839 raft_consensus.cc:3055] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:04.655823 12835 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:04.661082 12797 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:04.661320 12897 raft_consensus.cc:697] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:04.688774 12809 consensus_queue.cc:1243] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer 0d7efc590a1041a9bd4cd4a971332dfb log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260321 02:59:04.701054 11383 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 5.
I20260321 02:59:04.664690 12536 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:04.715706 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 113150us
I20260321 02:59:04.716723 11275 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:04.652945 12804 raft_consensus.cc:515] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:04.687076 12839 raft_consensus.cc:740] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
W20260321 02:59:04.727437 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:04.690263 12835 raft_consensus.cc:3060] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:04.715674 12833 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.965s user 0.000s sys 0.024s
I20260321 02:59:04.734302 11383 raft_consensus.cc:1275] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20260321 02:59:04.738343 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 57756.7us
I20260321 02:59:04.712754 12901 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:04.722407 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 92965.3us
I20260321 02:59:04.687045 12819 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:04.720099 12701 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:04.727802 12885 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:04.765079 11200 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:04.766695 11200 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:04.728993 12834 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:04.717592 12896 consensus_queue.cc:1059] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.027s
I20260321 02:59:04.733047 12783 raft_consensus.cc:697] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:04.776247 12901 consensus_queue.cc:1048] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:04.777357 12904 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:04.761377 11380 raft_consensus.cc:1275] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:59:04.768988 11200 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:04.776976 12783 consensus_queue.cc:237] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:04.758813 12839 consensus_queue.cc:260] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:04.745340 12833 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.003s user 0.000s sys 0.024s
I20260321 02:59:04.801517 12814 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:04.802426 12814 raft_consensus.cc:2749] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:04.804872 12804 leader_election.cc:290] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:04.805097 11200 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:04.756250 12811 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:04.785867 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 69225.7us
I20260321 02:59:04.777921 12901 raft_consensus.cc:2749] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:04.740562 12757 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:04.813726 11275 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:04.815261 11275 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:04.816599 11275 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:04.796833 12905 raft_consensus.cc:2804] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:04.799649 12898 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:04.805521 12804 raft_consensus.cc:2749] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:04.781454 12701 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:04.836669 11275 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:04.791936 12914 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:04.841974 12810 raft_consensus.cc:2804] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260321 02:59:04.845445 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 88152.3us
I20260321 02:59:04.829281 12839 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:04.826059 12811 raft_consensus.cc:2749] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:04.823786 12889 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:04.841467 11275 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:04.857857 11275 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:04.858997 11275 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:04.848873 11942 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:04.853281 12907 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:04.840325 12898 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:04.855346 12903 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:04.834553 12905 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
W20260321 02:59:04.856364 12911 log.cc:927] Time spent T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.128s user 0.001s sys 0.000s
I20260321 02:59:04.863838 12811 raft_consensus.cc:2749] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:04.845455 12906 raft_consensus.cc:2749] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:04.839920 12926 raft_consensus.cc:2749] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:04.856915 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:04.794011 12897 consensus_queue.cc:237] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:04.846163 12810 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:04.859869 12814 raft_consensus.cc:2749] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:04.872849 12905 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:04.869217 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:04.889593 11275 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:04.882501 12810 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:04.890239 11974 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 1.
I20260321 02:59:04.897082 12701 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:04.897430 12925 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:04.897776 12701 raft_consensus.cc:2804] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:04.898294 12701 raft_consensus.cc:697] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:04.898034 12925 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:04.901072 11276 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:04.850399 12914 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:04.920032 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:04.923254 11275 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:04.925277 11275 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:04.928865 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:04.902175 12819 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:04.919615 12924 raft_consensus.cc:2804] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:04.919876 12814 raft_consensus.cc:2749] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:04.899283 12701 consensus_queue.cc:237] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:04.926213 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:04.929106 12852 raft_consensus.cc:2749] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:04.933378 12906 raft_consensus.cc:2749] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:04.920096 12923 raft_consensus.cc:2804] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:04.938030 12924 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:04.954391 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 147854us
I20260321 02:59:04.939883 12811 consensus_queue.cc:1048] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260321 02:59:04.962823 11275 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:04.965098 12931 log.cc:927] Time spent T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260321 02:59:04.969838 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:04.951712 12923 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:04.956866 12924 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:04.971534 12818 consensus_queue.cc:1048] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260321 02:59:04.972400 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:04.980629 11303 raft_consensus.cc:1275] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260321 02:59:04.980755 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:04.981040 11303 pending_rounds.cc:85] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da: Aborting all ops after (but not including) 1
I20260321 02:59:04.981309 11303 pending_rounds.cc:107] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260321 02:59:04.981540 11308 raft_consensus.cc:2393] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 2: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
I20260321 02:59:04.977006 12923 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:04.982414 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:04.981566 11303 raft_consensus.cc:2889] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:04.972460 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:04.974184 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:04.974812 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:04.974857 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:04.986030 11236 raft_consensus.cc:2393] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:04.975095 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:04.976397 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:04.976730 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:04.987205 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:04.966544 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:04.968511 12814 raft_consensus.cc:2749] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:04.988850 11306 raft_consensus.cc:1275] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:59:04.988849 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:04.991788 11311 raft_consensus.cc:1275] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260321 02:59:04.988215 11305 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:04.957568 12934 raft_consensus.cc:2749] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260321 02:59:04.975998 12919 log.cc:927] Time spent T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20260321 02:59:04.993523 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:04.970595 12833 consensus_queue.cc:1048] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260321 02:59:04.994626 11234 raft_consensus.cc:1240] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 2. Current term is 3. Ops: []
I20260321 02:59:05.011780 12914 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:04.978452 12916 log.cc:927] Time spent T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.133s user 0.000s sys 0.001s
W20260321 02:59:04.999323 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:04.970992 12925 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:05.007114 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:05.027002 11311 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:05.007958 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:05.008620 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:05.028183 11305 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:04.978901 12898 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:05.028519 11303 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:05.011444 12929 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:04.962474 12852 consensus_queue.cc:1048] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260321 02:59:05.017288 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:05.038702 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:05.039645 11302 raft_consensus.cc:2468] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:59:05.010171 12889 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:05.047659 11302 raft_consensus.cc:1275] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:05.050473 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:05.050563 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:05.051385 11303 raft_consensus.cc:2468] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:59:05.051882 11383 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:05.054322 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:05.054586 11133 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44688 (request call id 127) took 1531 ms. Trace:
I20260321 02:59:05.055179 11309 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:05.030025 12913 raft_consensus.cc:2749] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285) with higher term. Message: Invalid argument: T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5.
I20260321 02:59:05.055111 11133 rpcz_store.cc:276] 0321 02:59:03.523272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:03.523438 (+ 166us) service_pool.cc:225] Handling call
0321 02:59:03.572021 (+ 48583us) spinlock_profiling.cc:243] Waited 3.71 ms on lock 0x7b54002d6780. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8afec0c91 00007fa8aff954a9 00007fa8aff8b0cd 00007fa8aff4c72b 00007fa8aff4c504 00007fa8b24169c1 00007fa8b241721f 00007fa8b22da45b 00007fa8b23dde9e 00007fa8b08d64a4 00007fa8b08d6432
0321 02:59:05.054523 (+1482502us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:59:03.577232 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:59:03.577851 (+ 619us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:59:03.577882 (+ 31us) write_op.cc:435] Acquired schema lock
0321 02:59:03.577900 (+ 18us) tablet.cc:661] Decoding operations
0321 02:59:03.578206 (+ 306us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:59:03.578256 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0321 02:59:03.578283 (+ 27us) tablet.cc:684] Acquiring locks for 1 operations
0321 02:59:03.580939 (+ 2656us) tablet.cc:700] Row locks acquired
0321 02:59:03.580960 (+ 21us) write_op.cc:260] PREPARE: finished
0321 02:59:03.581086 (+ 126us) write_op.cc:270] Start()
0321 02:59:03.581192 (+ 106us) write_op.cc:276] Timestamp: P: 1774061943581047 usec, L: 0
0321 02:59:03.581212 (+ 20us) op_driver.cc:348] REPLICATION: starting
0321 02:59:03.583264 (+ 2052us) log.cc:844] Serialized 526 byte log entry
0321 02:59:03.804819 (+221555us) op_driver.cc:464] REPLICATION: finished
0321 02:59:04.253729 (+448910us) write_op.cc:301] APPLY: starting
0321 02:59:04.257764 (+ 4035us) tablet.cc:1366] starting BulkCheckPresence
0321 02:59:04.273968 (+ 16204us) tablet.cc:1369] finished BulkCheckPresence
0321 02:59:04.309284 (+ 35316us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:59:04.358282 (+ 48998us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:59:04.385742 (+ 27460us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0321 02:59:04.388505 (+ 2763us) write_op.cc:312] APPLY: finished
0321 02:59:04.413363 (+ 24858us) log.cc:844] Serialized 23 byte log entry
0321 02:59:04.413790 (+ 427us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:04.413902 (+ 112us) write_op.cc:454] Released schema lock
0321 02:59:04.413962 (+ 60us) write_op.cc:341] FINISH: Updating metrics
0321 02:59:04.414484 (+ 522us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:04.414530 (+ 46us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":8525568,"child_traces":[["op",{"apply.queue_time_us":320298,"apply.run_cpu_time_us":2934,"apply.run_wall_time_us":195962,"num_ops":1,"prepare.queue_time_us":2988,"prepare.run_cpu_time_us":2600,"prepare.run_wall_time_us":6294,"replication_time_us":204884,"thread_start_us":117788,"threads_started":1}]]}
I20260321 02:59:05.060701 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:05.061962 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:05.062705 11383 raft_consensus.cc:2368] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5.
I20260321 02:59:05.068358 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:05.088495 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 77766.8us
I20260321 02:59:05.088696 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:05.093786 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:05.098014 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:05.101446 11379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1281) took 1420 ms. Trace:
W20260321 02:59:05.089813 12922 log.cc:927] Time spent T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.117s user 0.001s sys 0.000s
I20260321 02:59:05.109436 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:05.101977 11379 rpcz_store.cc:276] 0321 02:59:03.680702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:03.680922 (+ 220us) service_pool.cc:225] Handling call
0321 02:59:03.681240 (+ 318us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:03.739828 (+ 58588us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:03.739876 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:04.423361 (+683485us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b2c000d0e18. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8aae48c81 00007fa8aae585e2 00007fa8ac587de1 00007fa8ac54c81a 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22
0321 02:59:04.423751 (+ 390us) log.cc:844] Serialized 26 byte log entry
0321 02:59:04.612196 (+188445us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:59:04.612239 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:04.612330 (+ 91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:05.093879 (+481549us) raft_consensus.cc:1685] finished
0321 02:59:05.093912 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:05.101371 (+ 7459us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3680,"raft.queue_time_us":20150,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":13562,"spinlock_wait_cycles":51273472,"thread_start_us":245998,"threads_started":2,"wal-append.queue_time_us":291891}
I20260321 02:59:05.112028 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:05.115698 11303 raft_consensus.cc:3060] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:05.085857 11380 raft_consensus.cc:1275] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260321 02:59:05.092473 12818 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:05.123662 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:05.124539 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:05.127035 11379 raft_consensus.cc:3055] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:05.127359 11379 raft_consensus.cc:740] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:05.128666 11383 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:05.102384 12834 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:05.050055 12889 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:05.134152 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:05.122210 12939 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:05.133041 11379 consensus_queue.cc:260] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:05.024731 12938 consensus_queue.cc:1048] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.146s
I20260321 02:59:05.136776 11309 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:59:05.169652 12923 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:05.167132 12804 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:05.132601 12909 log.cc:927] Time spent T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.636s user 0.002s sys 0.004s
I20260321 02:59:05.138911 12834 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
W20260321 02:59:05.150549 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:05.171779 12838 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
W20260321 02:59:05.220165 12785 consensus_meta.cc:287] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.560s user 0.014s sys 0.000s
I20260321 02:59:05.220775 12785 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:05.222151 11230 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:05.233958 11349 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:05.241163 12923 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:05.130125 12818 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:05.241794 12928 consensus_queue.cc:1059] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.104s
I20260321 02:59:05.242655 12950 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:05.242218 12809 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:05.235661 11235 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 5.
I20260321 02:59:05.244833 11349 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:05.241657 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:05.055349 12929 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:05.249963 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 123298us
I20260321 02:59:05.258905 11305 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:05.175427 12905 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:05.217165 12885 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:05.238482 12720 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:05.262949 11275 connection.cc:232] RPC call timeout handler was delayed by -1.1921749269999964s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2056s for a response
I20260321 02:59:05.202247 12804 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:05.159590 12939 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:05.223556 12914 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:05.291822 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:05.295576 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:05.299649 11229 raft_consensus.cc:2393] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 4: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:05.299897 11376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40950 (request call id 973) took 1539 ms. Trace:
I20260321 02:59:05.300508 11376 rpcz_store.cc:276] 0321 02:59:03.760375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:04.325303 (+564928us) service_pool.cc:225] Handling call
0321 02:59:04.532027 (+206724us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:04.532701 (+ 674us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:05.299686 (+766985us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b54001d9a50. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54bc8d 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:05.299837 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":474687232}
I20260321 02:59:05.307219 12908 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:05.321898 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:05.322530 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:05.321436 11236 raft_consensus.cc:2468] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
I20260321 02:59:05.332584 11303 raft_consensus.cc:2468] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
W20260321 02:59:05.307612 12946 log.cc:927] Time spent T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.119s user 0.001s sys 0.000s
I20260321 02:59:05.342233 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:05.342893 11383 raft_consensus.cc:1275] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:05.342985 11275 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:05.291095 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:05.345695 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:05.359272 11229 raft_consensus.cc:2468] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
I20260321 02:59:05.302955 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:05.360915 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:05.344015 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:05.350329 12935 consensus_queue.cc:1048] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:05.350358 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:05.305325 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:05.353863 12944 consensus_queue.cc:1048] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:59:05.365710 11376 raft_consensus.cc:2376] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3. Re-sending same reply.
I20260321 02:59:05.351361 12730 raft_consensus.cc:2749] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:59:05.366724 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:05.291595 12809 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:05.334412 12785 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:05.372644 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:05.381668 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:05.381914 11229 raft_consensus.cc:2468] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
I20260321 02:59:05.288498 12950 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:05.376873 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:05.355304 12925 consensus_queue.cc:1048] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:59:05.382613 11376 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:05.347811 12811 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:05.385154 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:05.395036 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:05.308167 12914 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:05.401407 11231 raft_consensus.cc:1240] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 2. Current term is 3. Ops: []
I20260321 02:59:05.299932 12885 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:05.404575 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:05.419900 11376 raft_consensus.cc:2410] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 3 because replica has last-logged OpId of term: 2 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260321 02:59:05.368402 11972 raft_consensus.cc:3060] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:05.472077 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:05.412967 11383 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
I20260321 02:59:05.378562 11227 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
I20260321 02:59:05.340106 12908 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:05.493145 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:05.496697 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:05.496398 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:05.509359 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:05.510448 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:05.511704 11383 raft_consensus.cc:2393] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 5: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:05.413635 12900 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:05.517200 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:05.518877 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:05.396056 12889 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:05.415148 12923 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:05.522287 12809 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:05.523077 12908 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:05.523715 12923 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:05.523079 12809 raft_consensus.cc:2749] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:05.525477 12914 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:05.525964 12944 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:05.526381 12923 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:05.516937 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 210484us
I20260321 02:59:05.526659 12944 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:05.527762 11200 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:05.528893 11200 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:05.528954 12944 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:05.530155 11200 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:05.531034 11200 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:05.532568 11230 raft_consensus.cc:1240] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4. Ops: []
W20260321 02:59:05.535315 12899 consensus_meta.cc:287] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.945s user 0.008s sys 0.008s
I20260321 02:59:05.535899 12899 raft_consensus.cc:2749] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:05.539331 11378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40950 (request call id 976) took 1754 ms. Trace:
I20260321 02:59:05.539899 11378 rpcz_store.cc:276] 0321 02:59:03.784740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:04.410650 (+625910us) service_pool.cc:225] Handling call
0321 02:59:04.411123 (+ 473us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:04.590390 (+179267us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:05.538961 (+948571us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b54000c3d54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:05.539249 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":330162432}
I20260321 02:59:05.541543 12804 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:05.542346 12804 consensus_queue.cc:1048] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:05.543082 11385 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40950 (request call id 985) took 1752 ms. Trace:
I20260321 02:59:05.545125 12679 raft_consensus.cc:2749] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:05.545634 12819 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:05.545969 12679 raft_consensus.cc:2804] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:05.546263 12819 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:05.546561 12679 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:05.546864 12819 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:05.547111 12679 raft_consensus.cc:3060] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:05.550137 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:05.551127 11227 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
I20260321 02:59:05.543725 11385 rpcz_store.cc:276] 0321 02:59:03.790727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:04.427915 (+637188us) service_pool.cc:225] Handling call
0321 02:59:04.434161 (+ 6246us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:04.588703 (+154542us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:05.542729 (+954026us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b5400373554. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:05.543020 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":351497472}
I20260321 02:59:05.560077 11385 raft_consensus.cc:1275] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260321 02:59:05.515978 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 271196us
I20260321 02:59:05.413422 12834 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:05.415066 12811 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:05.573803 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:05.509368 12935 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:05.576160 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:05.582566 11381 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1293) took 1045 ms. Trace:
I20260321 02:59:05.584349 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:05.584965 12809 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:05.585505 12952 log.cc:927] Time spent T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20260321 02:59:05.588582 12811 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:05.585518 12809 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:05.595144 12809 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:05.568171 12908 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:05.596668 12908 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:05.598829 12738 raft_consensus.cc:2749] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:05.599272 12908 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:05.602002 11382 raft_consensus.cc:1275] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20260321 02:59:05.604310 11377 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40950 (request call id 966) took 1849 ms (client timeout 2056 ms). Trace:
W20260321 02:59:05.604882 11377 rpcz_store.cc:269] 0321 02:59:03.752314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:04.138837 (+386523us) service_pool.cc:225] Handling call
0321 02:59:05.601174 (+1462337us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b54000b9ad4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:05.601334 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":302250880}
I20260321 02:59:05.605880 12950 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:05.606812 12905 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:05.607717 12929 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:05.586882 12939 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:05.579631 11234 raft_consensus.cc:2393] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 4: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:05.572454 12804 raft_consensus.cc:2804] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:05.583205 11381 rpcz_store.cc:276] 0321 02:59:04.537318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:04.654342 (+117024us) service_pool.cc:225] Handling call
0321 02:59:04.752365 (+ 98023us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:04.933656 (+181291us) raft_consensus.cc:1534] Early marking committed up to index 2
0321 02:59:04.942103 (+ 8447us) log.cc:844] Serialized 14 byte log entry
0321 02:59:05.104741 (+162638us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:05.104795 (+ 54us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:05.104845 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:05.582094 (+477249us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b54001f68d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:05.582185 (+ 91us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:05.582448 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":10377,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":2605,"raft.queue_time_us":2218,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":11211,"spinlock_wait_cycles":416352000,"thread_start_us":162231,"threads_started":1,"wal-append.queue_time_us":260073}
I20260321 02:59:05.566809 11349 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:05.644995 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 78327.2us
I20260321 02:59:05.630722 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:05.645852 11349 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:05.576011 12934 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:05.647221 11349 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:05.630682 12804 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:05.657192 12804 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:05.658684 11384 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1292) took 1121 ms. Trace:
I20260321 02:59:05.659085 11384 rpcz_store.cc:276] 0321 02:59:04.537045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:04.653949 (+116904us) service_pool.cc:225] Handling call
0321 02:59:05.657822 (+1003873us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b58002e3298. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b1522680 00007fa8b150cfe9 00007fa8b150ca92 00007fa8aaa8ad34
0321 02:59:05.658095 (+ 273us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:05.658323 (+ 228us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:05.658349 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:05.658389 (+ 40us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:05.658417 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:05.658477 (+ 60us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:05.658616 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5070208}
I20260321 02:59:05.659996 11349 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:05.565104 12900 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:05.601598 12818 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:05.670305 11349 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:05.679433 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:05.669500 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 237273us
I20260321 02:59:05.645895 11227 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:05.646953 12809 raft_consensus.cc:2749] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:05.689881 11349 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:05.693549 12835 consensus_meta.cc:287] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.935s user 0.013s sys 0.002s
I20260321 02:59:05.694048 12835 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:05.696115 12835 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:05.684077 12896 consensus_queue.cc:1048] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260321 02:59:05.696826 12962 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:05.697290 12962 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:05.697613 12962 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:05.697759 12896 raft_consensus.cc:2804] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:05.698333 12896 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:05.671746 12934 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:05.669498 12904 raft_consensus.cc:2749] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:05.702116 11228 raft_consensus.cc:1240] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4. Ops: []
W20260321 02:59:05.702828 11275 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:05.703074 12785 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:05.703572 12785 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:05.705066 11382 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40950 (request call id 996) took 1896 ms. Trace:
I20260321 02:59:05.705574 11382 rpcz_store.cc:276] 0321 02:59:03.808498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:04.525351 (+716853us) service_pool.cc:225] Handling call
0321 02:59:05.601593 (+1076242us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:05.602439 (+ 846us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:05.704811 (+102372us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54000b9ad0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54bc8d 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:05.704992 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2406136192}
I20260321 02:59:05.706229 12785 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:05.708106 12839 consensus_meta.cc:287] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.852s user 0.019s sys 0.000s
I20260321 02:59:05.703763 11275 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:05.712399 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:05.713024 12811 raft_consensus.cc:2749] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:05.691514 12950 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:05.714457 11275 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:05.714960 12950 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:05.717594 12950 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:05.719257 12885 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:05.698750 12896 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:05.766253 11275 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:05.703815 12854 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:05.672322 12905 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:05.684595 12797 consensus_meta.cc:287] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 0.972s user 0.012s sys 0.004s
I20260321 02:59:05.667328 12961 raft_consensus.cc:2749] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:05.784960 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:05.786041 12900 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:05.782195 12852 raft_consensus.cc:2804] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:05.788221 12852 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:05.788733 12852 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:05.699750 12804 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:05.812453 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 44534.3us
I20260321 02:59:05.814483 11275 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:05.816061 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:05.782334 12810 consensus_meta.cc:287] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.874s user 0.009s sys 0.005s
I20260321 02:59:05.779369 12811 raft_consensus.cc:2804] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:05.609762 12935 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:05.825620 12730 raft_consensus.cc:2804] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:05.825510 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:05.828182 12811 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:05.793002 12797 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:05.811327 12905 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:05.823254 12908 consensus_queue.cc:1059] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.016s
I20260321 02:59:05.810645 12854 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:05.791558 12679 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:05.837890 12939 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:05.827562 12810 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:05.842024 12730 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:05.861340 11980 raft_consensus.cc:2368] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4.
I20260321 02:59:05.848832 12811 raft_consensus.cc:3060] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
W20260321 02:59:05.861819 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 330826us
W20260321 02:59:05.854200 12924 consensus_meta.cc:287] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.863s user 0.015s sys 0.006s
I20260321 02:59:05.881628 11200 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:05.901607 11306 raft_consensus.cc:1240] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: []
W20260321 02:59:05.909677 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 216414us
I20260321 02:59:05.909741 11384 raft_consensus.cc:1275] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260321 02:59:05.911209 11349 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:05.911767 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:05.918327 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:05.919107 11382 raft_consensus.cc:2468] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:05.921967 11349 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:05.864161 12730 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
W20260321 02:59:05.928164 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 46691.2us
I20260321 02:59:05.901069 12961 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:05.900831 12834 raft_consensus.cc:2749] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:05.869868 12938 consensus_queue.cc:1059] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.056s
I20260321 02:59:05.864293 12939 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:05.958652 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 137140us
W20260321 02:59:05.973266 11350 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.002s sys 0.029s
W20260321 02:59:05.975549 11350 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.557s user 0.003s sys 0.030s
I20260321 02:59:05.869024 12924 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:05.940929 12900 raft_consensus.cc:2804] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:05.982568 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:05.947460 12829 raft_consensus.cc:2804] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:05.954290 12839 raft_consensus.cc:2804] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:05.870957 12856 consensus_queue.cc:1048] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.076s
W20260321 02:59:05.993898 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 134623us
I20260321 02:59:05.988040 11275 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:05.997543 11304 raft_consensus.cc:2393] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 3: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:06.007812 11275 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:06.069178 11200 leader_election.cc:304] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:05.976169 12934 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:05.994235 12938 raft_consensus.cc:3055] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Stepping down as leader of term 3
W20260321 02:59:05.996227 12957 log.cc:927] Time spent T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.182s user 0.000s sys 0.001s
I20260321 02:59:05.991819 12900 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:06.072253 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 63466.9us
W20260321 02:59:06.022922 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:06.076855 11350 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 75120.9us
I20260321 02:59:05.992519 12829 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:05.996001 12955 log.cc:927] Time spent T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.171s user 0.002s sys 0.000s
I20260321 02:59:06.085007 11229 raft_consensus.cc:1240] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260321 02:59:06.091400 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:05.995280 12968 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:06.005937 12885 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
W20260321 02:59:06.101917 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 94195.6us
I20260321 02:59:06.006053 12877 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:06.107178 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.068710 12839 raft_consensus.cc:697] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:05.960259 12961 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.118453 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.122033 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.006157 12953 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:06.126699 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.122884 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:06.086798 12934 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:06.134052 11229 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:06.128752 11230 raft_consensus.cc:1240] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:06.080096 12938 raft_consensus.cc:740] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:06.097297 12900 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:06.084545 12976 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:06.089512 12785 raft_consensus.cc:2804] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:59:06.092005 12804 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:06.095513 12971 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:06.101215 12829 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:06.133864 12905 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:06.110671 12797 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:06.162220 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:06.128966 11227 raft_consensus.cc:1240] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:06.125008 12935 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:06.169263 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.148815 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:06.125248 12968 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.123624 12877 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.173559 12749 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:06.174540 12819 consensus_meta.cc:287] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.627s user 0.011s sys 0.002s
I20260321 02:59:06.163789 12977 raft_consensus.cc:2764] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260321 02:59:06.131817 12980 raft_consensus.cc:2749] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:06.182916 11229 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:06.220587 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.168859 11234 raft_consensus.cc:1240] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:06.148603 12854 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:06.197889 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 94003.1us
I20260321 02:59:06.197969 11200 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:06.201567 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.160964 12679 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:06.160888 12810 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:06.227342 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.163784 12985 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:06.228214 11230 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:06.152136 12934 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.237202 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.174178 12839 consensus_queue.cc:237] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.217234 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.176738 12785 raft_consensus.cc:697] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:06.227749 11200 leader_election.cc:400] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Vote denied by peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533) with higher term. Message: Invalid argument: T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5.
I20260321 02:59:06.129047 12885 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.192865 12799 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:06.194820 12939 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:06.238462 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:06.175508 12938 consensus_queue.cc:260] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.205051 12961 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:06.248411 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.137286 12953 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.303119 11307 raft_consensus.cc:2393] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 3: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:06.304527 11303 raft_consensus.cc:1240] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260321 02:59:06.310408 11230 raft_consensus.cc:2368] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5.
I20260321 02:59:06.238698 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:06.178078 12971 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.313694 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:06.307108 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:06.314863 11302 raft_consensus.cc:2468] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 2.
I20260321 02:59:06.314996 11229 raft_consensus.cc:1275] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260321 02:59:06.244179 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:06.242002 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:06.312511 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 84862.7us
I20260321 02:59:06.316340 11310 raft_consensus.cc:2410] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260321 02:59:06.313750 11230 raft_consensus.cc:1275] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260321 02:59:06.316632 11200 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:06.305977 11275 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:06.308502 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 219603us
I20260321 02:59:06.164605 12976 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:06.318248 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:06.318567 12990 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:06.318320 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:06.318256 11309 raft_consensus.cc:2393] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 4: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:06.249262 12757 raft_consensus.cc:2749] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:06.319854 11942 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:06.320101 11275 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:06.232102 12809 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:06.332996 11275 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:06.339146 11275 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:06.319119 11977 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
W20260321 02:59:06.249503 12962 consensus_meta.cc:287] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.551s user 0.013s sys 0.000s
I20260321 02:59:06.305593 12977 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:06.347926 11201 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:06.342001 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.356997 11231 raft_consensus.cc:2393] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 5: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:06.249759 12810 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:06.330967 11227 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:06.308082 12924 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:06.354002 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:06.353915 11275 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:06.320469 12970 log.cc:927] Time spent T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.325s user 0.000s sys 0.001s
I20260321 02:59:06.355413 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:06.332187 12992 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:06.316913 12961 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:06.223379 12819 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:06.332434 12972 log.cc:927] Time spent T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.339s user 0.000s sys 0.001s
I20260321 02:59:06.333487 12983 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:06.344858 12938 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:06.343453 12908 raft_consensus.cc:2749] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:06.345391 12715 raft_consensus.cc:2749] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:06.234560 12985 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.374241 12785 consensus_queue.cc:237] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:06.377245 11977 raft_consensus.cc:1275] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260321 02:59:06.377661 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:06.377826 11383 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1306) took 1363 ms. Trace:
W20260321 02:59:06.377940 12973 log.cc:927] Time spent T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.252s user 0.001s sys 0.001s
W20260321 02:59:06.365069 12965 log.cc:927] Time spent T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.283s user 0.000s sys 0.001s
I20260321 02:59:06.349599 12980 raft_consensus.cc:2749] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:06.376405 11309 raft_consensus.cc:1240] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:06.363797 12935 consensus_queue.cc:1048] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260321 02:59:06.308652 12799 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.369376 12977 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.366904 12964 raft_consensus.cc:2804] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:06.391921 11275 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:06.374253 11307 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:06.367290 12722 raft_consensus.cc:2804] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:06.418710 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:06.414242 11383 rpcz_store.cc:276] 0321 02:59:05.014556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:05.520738 (+506182us) service_pool.cc:225] Handling call
0321 02:59:05.527050 (+ 6312us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:05.527866 (+ 816us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:05.527897 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:05.528985 (+ 1088us) log.cc:844] Serialized 26 byte log entry
0321 02:59:05.860851 (+331866us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:59:05.860893 (+ 42us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:05.860967 (+ 74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:06.377605 (+516638us) raft_consensus.cc:1685] finished
0321 02:59:06.377626 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:06.377768 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10435,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":133,"thread_start_us":330140,"threads_started":1,"wal-append.queue_time_us":422517}
I20260321 02:59:06.352105 12809 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.422457 11307 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:06.362195 12962 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.427151 11227 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
W20260321 02:59:06.397403 12969 log.cc:927] Time spent T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.229s user 0.000s sys 0.001s
I20260321 02:59:06.366677 12810 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.366005 12757 consensus_queue.cc:1048] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20260321 02:59:06.405426 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:06.431905 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:06.443344 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:06.423094 12964 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:06.401841 12924 raft_consensus.cc:2749] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260321 02:59:06.401451 12981 log.cc:927] Time spent T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.187s user 0.000s sys 0.001s
I20260321 02:59:06.395066 12885 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:06.444355 11307 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:06.398097 12819 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:06.428812 12962 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:06.444376 11309 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:06.459038 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:06.425338 12722 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:06.462459 11308 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:06.371829 12991 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:06.453131 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:06.419540 12988 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:06.396646 12797 consensus_queue.cc:1048] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260321 02:59:06.467060 11307 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:06.488066 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:06.469285 11275 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:06.469074 12924 consensus_queue.cc:1048] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:06.406745 12961 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.453354 12964 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:06.420487 12983 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.444291 12908 raft_consensus.cc:2749] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:06.491677 12968 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:06.477231 12898 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:06.495616 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:06.484925 12877 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:06.480921 11302 raft_consensus.cc:2468] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:59:06.458460 12995 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:06.478212 12722 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:06.476836 12987 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:06.462368 12976 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:06.458236 12989 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:06.477545 12856 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:06.477049 12942 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:06.459878 11976 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Advancing to term 3
W20260321 02:59:06.513406 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 50745.5us
W20260321 02:59:06.513792 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 44608.3us
I20260321 02:59:06.389096 12992 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:06.525956 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:06.543426 11384 raft_consensus.cc:2410] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260321 02:59:06.486459 12991 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.459424 12980 consensus_queue.cc:1059] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260321 02:59:06.496246 12799 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:06.533187 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:06.534052 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:06.535138 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:06.498962 12935 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
W20260321 02:59:06.538113 11351 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.520s user 0.010s sys 0.011s
I20260321 02:59:06.556871 11942 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:06.559903 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:06.524402 11973 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45098 (request call id 266) took 2014 ms. Trace:
W20260321 02:59:06.539497 12811 consensus_meta.cc:287] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.667s user 0.012s sys 0.000s
I20260321 02:59:06.540719 11349 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:06.541213 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:06.520252 12934 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:06.558542 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:06.568884 11376 raft_consensus.cc:2376] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4. Re-sending same reply.
I20260321 02:59:06.563807 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.569418 11349 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:06.531553 12908 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:06.539466 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:06.536762 12971 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:06.575526 11349 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:06.523680 12898 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.569980 11231 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:06.584295 11349 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:06.570508 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:06.555388 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:06.586724 11349 leader_election.cc:400] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5.
I20260321 02:59:06.587237 11349 leader_election.cc:403] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20260321 02:59:06.596639 11348 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.005s sys 0.054s
W20260321 02:59:06.597278 11348 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.534s user 0.005s sys 0.054s
I20260321 02:59:06.567045 12968 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:06.540561 12856 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:06.598245 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:06.608270 11309 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 5
I20260321 02:59:06.583263 12962 raft_consensus.cc:2749] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:06.581179 12877 raft_consensus.cc:2749] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:06.529902 12995 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.550578 12987 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.588116 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.582646 12974 raft_consensus.cc:2749] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:06.632673 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.564284 11973 rpcz_store.cc:276] 0321 02:59:04.502954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:04.504954 (+ 2000us) service_pool.cc:225] Handling call
0321 02:59:04.576194 (+ 71240us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:04.629886 (+ 53692us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:04.634754 (+ 4868us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:04.657338 (+ 22584us) log.cc:844] Serialized 26 byte log entry
0321 02:59:04.800711 (+143373us) raft_consensus.cc:1645] Marking committed up to 5
0321 02:59:04.800805 (+ 94us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:04.800893 (+ 88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:04.843220 (+ 42327us) raft_consensus.cc:1685] finished
0321 02:59:04.843255 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:06.491761 (+1648506us) spinlock_profiling.cc:243] Waited 55.3 ms on lock 0x7b54006400d0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54bc8d 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:06.504175 (+ 12414us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":12170,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":12181,"spinlock_wait_cycles":127147904,"thread_start_us":130914,"threads_started":1,"wal-append.queue_time_us":175713,"wal-append.run_cpu_time_us":1646,"wal-append.run_wall_time_us":1008697}
I20260321 02:59:06.499047 12988 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:06.648439 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.643378 12961 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:06.600811 12852 consensus_meta.cc:287] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.812s user 0.009s sys 0.004s
I20260321 02:59:06.599932 11227 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:06.540318 12989 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.576833 12811 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.563513 12935 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:06.652627 11231 raft_consensus.cc:2368] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
W20260321 02:59:06.652383 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 87552us
I20260321 02:59:06.669382 11942 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:06.604197 12924 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:06.652899 12978 log.cc:927] Time spent T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.169s user 0.001s sys 0.000s
I20260321 02:59:06.645011 12985 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:06.607487 12797 consensus_queue.cc:1059] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260321 02:59:06.560505 12942 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:06.666787 12805 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:06.624423 13007 consensus_queue.cc:1048] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.099s
I20260321 02:59:06.702593 12805 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:06.652982 12809 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:06.627573 12968 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.709342 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:06.662002 12810 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:06.714784 12805 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:06.697762 12996 raft_consensus.cc:2749] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:06.710557 11234 raft_consensus.cc:2393] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:06.684074 12977 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:06.700700 12877 raft_consensus.cc:3060] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 5
I20260321 02:59:06.693186 12995 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:06.742977 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 156453us
I20260321 02:59:06.799566 11349 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:06.770085 12993 consensus_queue.cc:1048] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
W20260321 02:59:06.749624 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 63089.4us
I20260321 02:59:06.774025 11974 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45078 (request call id 320) took 1810 ms. Trace:
I20260321 02:59:06.700892 12962 consensus_queue.cc:1059] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260321 02:59:06.685477 12924 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:06.745965 11972 consensus_meta.cc:287] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 1.237s user 0.011s sys 0.011s
I20260321 02:59:06.722991 12987 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:06.765765 12896 consensus_meta.cc:287] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.954s user 0.015s sys 0.000s
I20260321 02:59:06.769551 12898 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:06.748409 12988 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:06.755021 12983 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:06.800817 11349 leader_election.cc:400] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:59:06.823153 11349 leader_election.cc:403] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260321 02:59:06.814574 11972 raft_consensus.cc:2468] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:06.825160 11200 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:06.683709 12998 raft_consensus.cc:2804] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:06.802459 12809 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:06.803058 12991 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:06.779583 12990 raft_consensus.cc:2749] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:06.678125 12852 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.833954 11349 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:06.795163 11227 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:06.828655 12998 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:06.837982 12829 consensus_meta.cc:287] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.643s user 0.004s sys 0.009s
I20260321 02:59:06.845994 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:06.838610 12992 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:06.839002 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.852502 12829 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.841776 11200 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:06.842738 12991 raft_consensus.cc:2749] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:06.843212 12989 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:06.839643 11972 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45078 (request call id 322) took 1593 ms (client timeout 1541 ms). Trace:
I20260321 02:59:06.822394 12856 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:06.846702 11376 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:06.830854 12983 raft_consensus.cc:3060] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 3
I20260321 02:59:06.833491 12809 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:06.855232 11200 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:06.854022 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.856588 11235 raft_consensus.cc:1240] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3. Ops: []
I20260321 02:59:06.857344 12829 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:06.840272 12805 raft_consensus.cc:2804] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:06.859469 11311 raft_consensus.cc:1240] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260321 02:59:06.818213 12896 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.877354 11200 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:06.844663 12811 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:06.850930 12935 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:06.846333 12998 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:06.906884 11200 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:06.879673 12809 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:06.881697 12971 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:06.881639 12968 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:06.858907 11972 rpcz_store.cc:269] 0321 02:59:05.237972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:05.261078 (+ 23106us) service_pool.cc:225] Handling call
0321 02:59:06.830223 (+1569145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:06.810472 11974 rpcz_store.cc:276] 0321 02:59:04.961611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:04.965432 (+ 3821us) service_pool.cc:225] Handling call
0321 02:59:06.680081 (+1714649us) spinlock_profiling.cc:243] Waited 1.18 ms on lock 0x7b58002c3798. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b1522680 00007fa8b150cfe9 00007fa8b150ca92 00007fa8aaa8ad34
0321 02:59:06.708400 (+ 28319us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:06.738727 (+ 30327us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:06.742171 (+ 3444us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:06.749686 (+ 7515us) raft_consensus.cc:1645] Marking committed up to 5
0321 02:59:06.752642 (+ 2956us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:06.761805 (+ 9163us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:06.771583 (+ 9778us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2702592}
I20260321 02:59:06.910585 12982 raft_consensus.cc:2764] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260321 02:59:06.913429 12730 consensus_meta.cc:287] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.969s user 0.012s sys 0.001s
I20260321 02:59:06.917197 11309 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:06.918375 11200 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:06.883215 12805 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:06.922094 11305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49786 (request call id 105) took 1556 ms. Trace:
I20260321 02:59:06.937901 11305 rpcz_store.cc:276] 0321 02:59:05.365468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:05.367073 (+ 1605us) service_pool.cc:225] Handling call
0321 02:59:05.367352 (+ 279us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:05.591548 (+224196us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:05.591596 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:05.591658 (+ 62us) raft_consensus.cc:1645] Marking committed up to 5
0321 02:59:05.591702 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:06.921596 (+1329894us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b540035cad4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:06.921661 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:06.922020 (+ 359us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":515044096}
W20260321 02:59:06.923010 11236 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58214 (request call id 1018) took 1632 ms (client timeout 1666 ms). Trace:
W20260321 02:59:06.939364 11236 rpcz_store.cc:269] 0321 02:59:05.290531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:05.290830 (+ 299us) service_pool.cc:225] Handling call
0321 02:59:06.922833 (+1632003us) spinlock_profiling.cc:243] Waited 21.9 ms on lock 0x7b540019d7d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:06.922949 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":50433920}
I20260321 02:59:06.884312 11234 raft_consensus.cc:2376] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4. Re-sending same reply.
I20260321 02:59:06.917482 12935 raft_consensus.cc:2804] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:06.944379 12935 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:06.917191 12971 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:06.921029 12829 raft_consensus.cc:2804] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:06.903357 12947 raft_consensus.cc:2804] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:06.939379 12942 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:06.937287 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:06.940826 11306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1313) took 1843 ms. Trace:
I20260321 02:59:06.968294 11306 rpcz_store.cc:276] 0321 02:59:05.097787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:05.099768 (+ 1981us) service_pool.cc:225] Handling call
0321 02:59:05.100244 (+ 476us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:05.902091 (+801847us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:06.940642 (+1038551us) spinlock_profiling.cc:243] Waited 801 ms on lock 0x7b54003c82d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:06.940783 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1842445056}
I20260321 02:59:06.969926 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:06.971278 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:06.943320 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.931926 12730 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:06.946889 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:06.910812 13000 consensus_queue.cc:1059] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.125s
I20260321 02:59:06.955375 12971 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:06.945360 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.950548 12935 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:06.976725 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:06.982249 11306 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:06.961107 12997 raft_consensus.cc:2804] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:06.966661 12947 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:06.957080 12829 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:06.974264 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:07.011315 11305 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:06.985210 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:07.018296 11306 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:59:06.983021 11309 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:06.977038 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:07.035622 11381 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:06.941994 12964 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:06.991241 12852 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:06.995910 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:06.999310 11976 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:06.999854 12997 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:07.002547 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:06.964548 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 252125us
I20260321 02:59:06.958138 12924 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:07.000861 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:06.987651 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:07.059608 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:07.121136 11377 raft_consensus.cc:1240] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260321 02:59:07.121284 11306 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:07.108857 11231 raft_consensus.cc:1240] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:07.103896 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:07.122321 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:07.124356 11309 raft_consensus.cc:2393] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 3: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
W20260321 02:59:07.125399 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 58071us
W20260321 02:59:07.164645 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:07.166756 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:07.162528 11942 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:07.134404 13000 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
W20260321 02:59:07.101293 12749 consensus_meta.cc:287] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.892s user 0.015s sys 0.000s
I20260321 02:59:07.012347 12947 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:07.061260 13013 log.cc:927] Time spent T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
I20260321 02:59:07.059221 12925 raft_consensus.cc:2749] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:07.197991 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:07.051807 12805 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:07.053153 12829 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:07.196400 11233 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:07.117692 12896 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:07.167649 11975 raft_consensus.cc:3060] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:07.134775 12997 raft_consensus.cc:3060] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:07.199019 11229 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:07.200417 13019 consensus_queue.cc:1059] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260321 02:59:07.134646 12897 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:07.191638 12933 raft_consensus.cc:2804] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:07.174937 13020 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:07.195688 13000 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:07.223860 13020 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.225754 13000 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:07.219786 12933 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:07.226298 12998 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.226826 12933 raft_consensus.cc:3060] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:07.230094 12877 raft_consensus.cc:2749] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5.
I20260321 02:59:07.218016 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:07.230957 12998 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:07.231734 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:07.232815 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:07.175135 11234 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:59:07.197907 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:07.236495 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:07.209450 12818 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:07.240274 11381 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:07.244365 11305 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:07.245803 11942 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:07.225826 13020 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:07.258291 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:07.259093 11381 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
W20260321 02:59:07.260850 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 239645us
I20260321 02:59:07.233284 12897 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:07.269922 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:07.262333 11974 raft_consensus.cc:3055] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:59:07.258525 12942 raft_consensus.cc:2804] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:07.274044 11381 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:07.257620 12730 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:07.243795 12818 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.280182 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:07.281687 11349 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:07.298910 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:07.300081 11377 raft_consensus.cc:2468] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:59:07.301170 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:07.302452 11307 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:07.310539 11349 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:07.311232 11381 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:07.312451 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:07.313202 11381 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:07.314231 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:07.315554 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:07.277946 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:07.323163 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:07.276258 11974 raft_consensus.cc:740] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
W20260321 02:59:07.300747 12900 consensus_meta.cc:287] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 1.131s user 0.010s sys 0.014s
I20260321 02:59:07.286651 12964 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:07.284729 12856 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:07.339144 11384 raft_consensus.cc:2468] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:07.300000 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:07.309234 12942 raft_consensus.cc:697] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:07.326159 12738 raft_consensus.cc:2804] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:07.349275 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:07.362223 11381 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:07.350798 12809 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.354717 11975 raft_consensus.cc:2410] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260321 02:59:07.333742 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:07.328264 13023 raft_consensus.cc:2804] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:59:07.356623 11974 consensus_queue.cc:260] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:07.297994 12935 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.356940 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
W20260321 02:59:07.363157 12908 consensus_meta.cc:287] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.784s user 0.017s sys 0.008s
I20260321 02:59:07.362035 12738 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:07.382329 12897 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:07.383320 12935 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:07.385962 11377 raft_consensus.cc:2393] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 5: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:07.396618 11228 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58214 (request call id 1022) took 2038 ms. Trace:
W20260321 02:59:07.401672 12722 consensus_meta.cc:287] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.866s user 0.018s sys 0.001s
I20260321 02:59:07.400920 11228 rpcz_store.cc:276] 0321 02:59:05.358444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:05.358634 (+ 190us) service_pool.cc:225] Handling call
0321 02:59:05.359038 (+ 404us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:05.702558 (+343520us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:07.396356 (+1693798us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b540019dcd4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:07.396549 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":788534528}
I20260321 02:59:07.402060 12722 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.403147 11305 raft_consensus.cc:1240] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260321 02:59:07.401576 11304 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56630 (request call id 1305) took 2385 ms (client timeout 1670 ms). Trace:
I20260321 02:59:07.373664 12964 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:07.345219 12900 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.396618 12738 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:07.360064 12856 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:07.395591 11381 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:07.373471 13023 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:07.377249 11972 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:07.408706 13028 log.cc:927] Time spent T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20260321 02:59:07.409740 12722 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:07.385206 11975 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45078 (request call id 325) took 1681 ms (client timeout 1924 ms). Trace:
W20260321 02:59:07.404682 11304 rpcz_store.cc:269] 0321 02:59:05.016718 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:05.016926 (+ 208us) service_pool.cc:225] Handling call
0321 02:59:07.401327 (+2384401us) spinlock_profiling.cc:243] Waited 959 ms on lock 0x7b540020ba54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:07.401501 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2205786880}
I20260321 02:59:07.451480 11376 raft_consensus.cc:1240] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:07.412009 12809 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:07.442886 12856 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:07.415592 12834 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:07.446411 12829 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.449465 12947 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:07.397773 12908 raft_consensus.cc:2749] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:07.413760 11974 raft_consensus.cc:3060] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:07.443006 12805 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.452802 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:07.414234 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:07.438575 13023 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:07.415553 12897 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:07.460075 12834 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:07.473346 11381 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:07.473567 11303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1318) took 1118 ms. Trace:
I20260321 02:59:07.475311 11302 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:07.452558 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:07.479492 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:07.479811 11232 raft_consensus.cc:2368] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4.
I20260321 02:59:07.480120 11381 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:07.481230 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:07.415773 12964 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.479019 11303 rpcz_store.cc:276] 0321 02:59:06.355342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:06.355568 (+ 226us) service_pool.cc:225] Handling call
0321 02:59:06.378334 (+ 22766us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:06.437615 (+ 59281us) raft_consensus.cc:1534] Early marking committed up to index 3
0321 02:59:06.513054 (+ 75439us) log.cc:844] Serialized 14 byte log entry
0321 02:59:06.516050 (+ 2996us) log.cc:844] Serialized 14 byte log entry
0321 02:59:06.555240 (+ 39190us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:06.555295 (+ 55us) raft_consensus.cc:1645] Marking committed up to 3
0321 02:59:06.555863 (+ 568us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:07.473329 (+917466us) spinlock_profiling.cc:243] Waited 59 ms on lock 0x7b54003aeb54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:07.473375 (+ 46us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:07.473513 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18312,"prepare.queue_time_us":144516,"raft.queue_time_us":109295,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":21330,"spinlock_wait_cycles":200890112,"thread_start_us":97499,"threads_started":4,"wal-append.queue_time_us":69203}
I20260321 02:59:07.485078 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:07.495041 12834 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:07.495775 11227 raft_consensus.cc:2393] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:07.496228 11979 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:07.496953 11979 raft_consensus.cc:2368] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5.
I20260321 02:59:07.498785 11350 leader_election.cc:400] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285) with higher term. Message: Invalid argument: T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5.
I20260321 02:59:07.453434 12933 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:07.495029 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
W20260321 02:59:07.451696 11975 rpcz_store.cc:269] 0321 02:59:05.703477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:07.000518 (+1297041us) service_pool.cc:225] Handling call
0321 02:59:07.380028 (+379510us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260321 02:59:07.503772 13027 log.cc:927] Time spent T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.087s user 0.000s sys 0.001s
I20260321 02:59:07.505532 11979 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:07.506098 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:07.504256 11978 raft_consensus.cc:2376] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5. Re-sending same reply.
I20260321 02:59:07.502976 11234 raft_consensus.cc:2368] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5.
I20260321 02:59:07.509447 11303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1331) took 1085 ms. Trace:
W20260321 02:59:07.500201 13035 log.cc:927] Time spent T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260321 02:59:07.440845 12942 consensus_queue.cc:237] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:07.507741 12947 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:07.518150 12829 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:07.517177 12933 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:07.511004 11201 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:07.511034 11350 leader_election.cc:304] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:07.513345 11978 raft_consensus.cc:1240] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 2. Current term is 3. Ops: []
I20260321 02:59:07.513428 11303 rpcz_store.cc:276] 0321 02:59:06.424266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:07.492251 (+1067985us) service_pool.cc:225] Handling call
0321 02:59:07.508914 (+ 16663us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:07.509101 (+ 187us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:07.509135 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:07.509180 (+ 45us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:07.509208 (+ 28us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:07.509272 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:07.509396 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260321 02:59:07.515208 12971 consensus_meta.cc:287] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.527s user 0.007s sys 0.003s
I20260321 02:59:07.515604 11230 raft_consensus.cc:1240] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:07.524439 12877 raft_consensus.cc:2749] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:07.525207 12987 raft_consensus.cc:3055] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:07.525290 11201 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:07.525756 12987 raft_consensus.cc:740] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:07.526649 12987 consensus_queue.cc:260] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.527017 12736 raft_consensus.cc:2749] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:07.527187 12947 consensus_queue.cc:1059] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260321 02:59:07.528673 12736 raft_consensus.cc:3055] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:59:07.529079 12736 raft_consensus.cc:740] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:07.529183 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:07.530517 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:07.532164 11349 leader_election.cc:400] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5.
I20260321 02:59:07.532486 11349 leader_election.cc:403] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260321 02:59:07.527716 12987 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:07.525426 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:07.529983 12736 consensus_queue.cc:260] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:07.542161 11378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40950 (request call id 1018) took 1016 ms. Trace:
I20260321 02:59:07.537480 12968 consensus_queue.cc:1059] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:59:07.538430 11972 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:07.539516 12953 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:07.511483 11971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:07.542620 11378 rpcz_store.cc:276] 0321 02:59:06.525249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:06.527535 (+ 2286us) service_pool.cc:225] Handling call
0321 02:59:06.589925 (+ 62390us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:06.881742 (+291817us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:06.883901 (+ 2159us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:06.921835 (+ 37934us) log.cc:844] Serialized 26 byte log entry
0321 02:59:07.219391 (+297556us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:07.260110 (+ 40719us) log.cc:844] Serialized 14 byte log entry
0321 02:59:07.278343 (+ 18233us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:07.281967 (+ 3624us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:07.541911 (+259944us) raft_consensus.cc:1685] finished
0321 02:59:07.541934 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:07.542098 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39977,"prepare.queue_time_us":16722,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":62991,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":19830,"thread_start_us":439883,"threads_started":2,"wal-append.queue_time_us":330100}
I20260321 02:59:07.552628 12962 raft_consensus.cc:3055] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:07.553329 12962 raft_consensus.cc:740] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:07.553413 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:07.554204 12962 consensus_queue.cc:260] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.555218 12962 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:07.541008 12877 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:07.488194 12897 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:07.534546 13034 log.cc:927] Time spent T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260321 02:59:07.567301 12738 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:07.569195 12994 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260321 02:59:07.542430 12736 raft_consensus.cc:3060] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:07.569496 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:07.569876 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:07.570581 11380 raft_consensus.cc:2468] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
I20260321 02:59:07.571281 11380 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40950 (request call id 1023) took 1037 ms. Trace:
I20260321 02:59:07.545518 11303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1333) took 1082 ms. Trace:
I20260321 02:59:07.560763 11201 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:07.576637 11303 rpcz_store.cc:276] 0321 02:59:06.463580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:07.520967 (+1057387us) service_pool.cc:225] Handling call
0321 02:59:07.545022 (+ 24055us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:07.545202 (+ 180us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:07.545236 (+ 34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:07.545280 (+ 44us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:07.545309 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:07.545358 (+ 49us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:07.545465 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:07.543231 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:07.572188 12805 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:07.562824 12900 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:07.572695 11378 raft_consensus.cc:2393] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 6: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
W20260321 02:59:07.573992 11980 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45078 (request call id 323) took 2244 ms (client timeout 1918 ms). Trace:
W20260321 02:59:07.575591 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:07.533061 12829 raft_consensus.cc:2804] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:07.580054 11311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56616 (request call id 1449) took 1338 ms. Trace:
I20260321 02:59:07.532934 12971 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:07.583218 11308 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56616 (request call id 1451) took 1341 ms (client timeout 1706 ms). Trace:
I20260321 02:59:07.590404 11311 rpcz_store.cc:276] 0321 02:59:06.241250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:06.241388 (+ 138us) service_pool.cc:225] Handling call
0321 02:59:06.241774 (+ 386us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:06.881958 (+640184us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:07.579623 (+697665us) spinlock_profiling.cc:243] Waited 617 ms on lock 0x7b540018b354. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:07.579913 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1419784064}
W20260321 02:59:07.592669 11308 rpcz_store.cc:269] 0321 02:59:06.241850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:06.243789 (+ 1939us) service_pool.cc:225] Handling call
0321 02:59:07.582956 (+1339167us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b54003386d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:07.583137 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":336793344}
W20260321 02:59:07.588306 11980 rpcz_store.cc:269] 0321 02:59:05.329574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:05.329798 (+ 224us) service_pool.cc:225] Handling call
0321 02:59:07.573673 (+2243875us) spinlock_profiling.cc:243] Waited 499 ms on lock 0x7b540063ef54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:07.573868 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1148159872}
I20260321 02:59:07.596742 12783 raft_consensus.cc:3055] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:59:07.597208 12783 raft_consensus.cc:740] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:07.575538 11380 rpcz_store.cc:276] 0321 02:59:06.533934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:06.536867 (+ 2933us) service_pool.cc:225] Handling call
0321 02:59:07.569767 (+1032900us) spinlock_profiling.cc:243] Waited 4.92 ms on lock 0x7b58002e3298. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a94791ce 000000000035a3c9 00007fa8b14978e0 00007fa8b1560784 00007fa8b15607f1 00007fa8b150e760 00007fa8b150e044 00007fa8aaa8f834 00007fa8aaa8f7c2
0321 02:59:07.571201 (+ 1434us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11322240}
I20260321 02:59:07.597714 12738 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:07.598034 12783 consensus_queue.cc:260] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.598498 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:07.603418 11380 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:07.598713 12829 raft_consensus.cc:697] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:07.606101 11305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1327) took 1226 ms. Trace:
I20260321 02:59:07.606616 11305 rpcz_store.cc:276] 0321 02:59:06.379853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:07.250703 (+870850us) service_pool.cc:225] Handling call
0321 02:59:07.251057 (+ 354us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:07.403489 (+152432us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:07.605825 (+202336us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b54001ad6d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:07.606020 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":349205248}
I20260321 02:59:07.602347 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:07.585500 12953 raft_consensus.cc:2804] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:07.615738 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:07.614497 12983 consensus_meta.cc:287] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.735s user 0.017s sys 0.006s
I20260321 02:59:07.630865 12983 raft_consensus.cc:2749] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:59:07.631421 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:07.632109 11234 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:07.633616 12997 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.625761 11980 pending_rounds.cc:85] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56: Aborting all ops after (but not including) 4
I20260321 02:59:07.637610 11229 raft_consensus.cc:2393] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 3: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:07.637852 11980 pending_rounds.cc:107] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260321 02:59:07.638173 11980 raft_consensus.cc:2889] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:07.639829 11381 raft_consensus.cc:2468] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:07.599478 12783 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:07.629529 11200 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:07.644763 11227 raft_consensus.cc:2393] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:07.645151 12971 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:07.623821 12953 raft_consensus.cc:697] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
W20260321 02:59:07.646970 11233 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58200 (request call id 1414) took 1530 ms (client timeout 1963 ms). Trace:
I20260321 02:59:07.646459 12953 consensus_queue.cc:237] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:07.652994 11302 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56630 (request call id 1326) took 1281 ms. Trace:
I20260321 02:59:07.653448 11302 rpcz_store.cc:276] 0321 02:59:06.371466 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:06.520358 (+148892us) service_pool.cc:225] Handling call
0321 02:59:07.652777 (+1132419us) spinlock_profiling.cc:243] Waited 900 ms on lock 0x7b54000e79d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:07.652928 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2070955776}
I20260321 02:59:07.654532 11383 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40950 (request call id 1017) took 1129 ms. Trace:
I20260321 02:59:07.655498 12934 raft_consensus.cc:3055] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:59:07.641767 12947 consensus_queue.cc:1059] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260321 02:59:07.655920 12934 raft_consensus.cc:740] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:07.657810 13008 consensus_queue.cc:1048] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:07.658187 12947 raft_consensus.cc:3055] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:07.658726 12947 raft_consensus.cc:740] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
W20260321 02:59:07.657382 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 1.01806e+06us
I20260321 02:59:07.659914 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:07.647548 11233 rpcz_store.cc:269] 0321 02:59:06.116677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:06.118101 (+ 1424us) service_pool.cc:225] Handling call
0321 02:59:07.646548 (+1528447us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540037a354. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:07.646867 (+ 319us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2462110080}
I20260321 02:59:07.655117 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:07.659969 11275 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:07.658043 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 124256us
I20260321 02:59:07.648401 12997 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:07.682461 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 53054.8us
I20260321 02:59:07.655102 11383 rpcz_store.cc:276] 0321 02:59:06.524927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:06.525077 (+ 150us) service_pool.cc:225] Handling call
0321 02:59:06.525405 (+ 328us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:06.525864 (+ 459us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:06.525894 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:06.532195 (+ 6301us) log.cc:844] Serialized 26 byte log entry
0321 02:59:06.660580 (+128385us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:06.869566 (+208986us) log.cc:844] Serialized 14 byte log entry
0321 02:59:06.878658 (+ 9092us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:06.878784 (+ 126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:07.652319 (+773535us) spinlock_profiling.cc:243] Waited 1.99 ms on lock 0x7b1800173ce0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b2300441 00007fa8b230cca1 00007fa8a94b61c3 00007fa8a94b8591 00007fa8a94b8549 00007fa8a94b84d9 00007fa8a94b84a1 00007fa8a94b779d 0000000000379254 0000000000379099 00007fa8a949974a
0321 02:59:07.654111 (+ 1792us) raft_consensus.cc:1685] finished
0321 02:59:07.654236 (+ 125us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:07.654455 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16210,"prepare.queue_time_us":338381,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":10259,"raft.run_cpu_time_us":372,"raft.run_wall_time_us":30649,"spinlock_wait_cycles":4571392,"thread_start_us":257964,"threads_started":2,"wal-append.queue_time_us":345353}
I20260321 02:59:07.683609 11200 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:07.694109 12829 consensus_queue.cc:237] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:07.696343 13033 log.cc:927] Time spent T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.210s user 0.000s sys 0.004s
I20260321 02:59:07.642522 12964 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:07.681453 12990 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:07.699023 12990 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.700488 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:07.704768 11306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1320) took 1348 ms. Trace:
W20260321 02:59:07.672536 13039 log.cc:927] Time spent T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20260321 02:59:07.704808 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:07.705209 11306 rpcz_store.cc:276] 0321 02:59:06.356281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:07.125036 (+768755us) service_pool.cc:225] Handling call
0321 02:59:07.125342 (+ 306us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:07.125586 (+ 244us) raft_consensus.cc:1534] Early marking committed up to index 3
0321 02:59:07.269394 (+143808us) log.cc:844] Serialized 14 byte log entry
0321 02:59:07.704278 (+434884us) log.cc:844] Serialized 14 byte log entry
0321 02:59:07.704440 (+ 162us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:07.704495 (+ 55us) raft_consensus.cc:1645] Marking committed up to 3
0321 02:59:07.704535 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:07.704601 (+ 66us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:07.704706 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":346107,"prepare.queue_time_us":237351,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"thread_start_us":179182,"threads_started":2,"wal-append.queue_time_us":129034}
I20260321 02:59:07.719694 11200 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:07.721518 11236 raft_consensus.cc:1240] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:07.724305 11304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1329) took 1310 ms. Trace:
I20260321 02:59:07.724864 11200 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:07.725246 11236 raft_consensus.cc:1275] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260321 02:59:07.714323 12785 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.947s user 0.016s sys 0.062s
W20260321 02:59:07.727499 12785 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.963s user 0.016s sys 0.062s
I20260321 02:59:07.728636 11234 raft_consensus.cc:1240] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260321 02:59:07.722929 12950 raft_consensus.cc:2749] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:07.731557 12971 raft_consensus.cc:2749] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:07.733837 12803 raft_consensus.cc:2804] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:07.733908 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:07.734030 11200 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:07.734431 12803 raft_consensus.cc:697] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:07.734752 11382 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:07.735162 11200 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:07.735297 12803 consensus_queue.cc:237] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.659664 12947 consensus_queue.cc:260] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:07.737043 13043 log.cc:927] Time spent T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.120s user 0.000s sys 0.001s
I20260321 02:59:07.737107 11200 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:07.739912 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:07.735641 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:07.744688 11383 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:07.741062 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:07.745687 11381 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
W20260321 02:59:07.746634 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 86788.9us
I20260321 02:59:07.747517 11200 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:07.747519 11275 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:07.750669 11349 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:07.704582 12998 raft_consensus.cc:3055] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:07.705595 12997 raft_consensus.cc:2749] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:07.705248 13006 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:07.763818 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:07.765915 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:07.766626 11381 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:07.737303 12947 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:07.754382 13004 raft_consensus.cc:2804] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:59:07.746929 12848 raft_consensus.cc:2804] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:07.747886 12720 raft_consensus.cc:2804] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:07.691274 12989 consensus_queue.cc:1048] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
I20260321 02:59:07.773386 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:07.762703 13016 raft_consensus.cc:2804] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:07.761649 12998 raft_consensus.cc:740] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:07.765964 12953 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:07.780710 12924 raft_consensus.cc:2804] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:07.758994 12856 raft_consensus.cc:2804] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:59:07.658756 12934 consensus_queue.cc:260] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.781143 13023 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.802436 12877 raft_consensus.cc:2749] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5.
I20260321 02:59:07.793941 12990 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:07.803109 13023 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:07.780604 12848 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:07.781625 13004 raft_consensus.cc:697] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:07.769438 12897 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:07.781530 12720 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:07.809686 12848 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:07.792034 13016 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:07.800028 12953 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:07.798931 12924 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:07.772202 13006 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.812885 12856 raft_consensus.cc:697] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:07.821779 12953 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:07.822115 12924 raft_consensus.cc:3060] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:07.820201 13016 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:07.831334 13004 consensus_queue.cc:237] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:07.822469 12998 consensus_queue.cc:260] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.840975 12998 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:07.843641 13016 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.846637 13016 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:07.888190 11200 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:07.893612 11303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1340) took 1169 ms. Trace:
W20260321 02:59:07.894371 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:07.894986 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:07.894266 11303 rpcz_store.cc:276] 0321 02:59:06.723588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:07.578272 (+854684us) service_pool.cc:225] Handling call
0321 02:59:07.578885 (+ 613us) raft_consensus.cc:1471] Updating replica for 2 ops
0321 02:59:07.724137 (+145252us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:07.724168 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0321 02:59:07.724754 (+ 586us) log.cc:844] Serialized 52 byte log entry
0321 02:59:07.892771 (+168017us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:59:07.892865 (+ 94us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:07.892973 (+ 108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:07.893047 (+ 74us) raft_consensus.cc:1685] finished
0321 02:59:07.893063 (+ 16us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:07.893289 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":157918,"raft.queue_time_us":146557,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":18491,"thread_start_us":155077,"threads_started":1,"wal-append.queue_time_us":90399}
W20260321 02:59:07.893698 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 129680us
I20260321 02:59:07.903733 11302 raft_consensus.cc:1240] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:07.912609 11385 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
W20260321 02:59:07.788910 11940 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:07.816025 12720 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:07.920377 12749 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:07.921398 12749 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:07.922379 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:07.921306 11349 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:07.926209 12938 consensus_queue.cc:1059] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260321 02:59:07.937774 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:07.941440 11973 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45088 (request call id 206) took 1262 ms (client timeout 1502 ms). Trace:
W20260321 02:59:07.942308 11973 rpcz_store.cc:269] 0321 02:59:06.679324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:06.680259 (+ 935us) service_pool.cc:225] Handling call
0321 02:59:07.921948 (+1241689us) spinlock_profiling.cc:243] Waited 6.87 ms on lock 0x7b58002c3798. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a94791ce 000000000035a3c9 00007fa8b14978e0 00007fa8b1560784 00007fa8b15607f1 00007fa8b150e760 00007fa8b150e044 00007fa8aaa8f834 00007fa8aaa8f7c2
0321 02:59:07.941325 (+ 19377us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15792896}
I20260321 02:59:07.944242 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:07.956346 11235 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58200 (request call id 1411) took 1841 ms. Trace:
I20260321 02:59:07.942114 11380 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:07.957314 12856 consensus_queue.cc:237] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.960152 11229 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58200 (request call id 1435) took 1020 ms. Trace:
I20260321 02:59:07.960614 11229 rpcz_store.cc:276] 0321 02:59:06.939866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:07.232597 (+292731us) service_pool.cc:225] Handling call
0321 02:59:07.959635 (+727038us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x7b540019d7d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:07.959902 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":929233792}
I20260321 02:59:07.964365 11377 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:07.971292 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:07.980366 11305 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
W20260321 02:59:07.975757 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:07.970026 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:07.976620 11302 raft_consensus.cc:1240] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:07.978274 12749 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:07.956941 11235 rpcz_store.cc:276] 0321 02:59:06.115393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:06.116907 (+ 1514us) service_pool.cc:225] Handling call
0321 02:59:06.117375 (+ 468us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:06.876579 (+759204us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:07.955675 (+1079096us) spinlock_profiling.cc:243] Waited 739 ms on lock 0x7b5400379e54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:07.956107 (+ 432us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1699199744}
I20260321 02:59:07.976146 12962 consensus_queue.cc:1048] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:07.960179 13016 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:07.995496 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:07.995311 13016 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:07.999111 13016 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:08.010737 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.008673 11303 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
W20260321 02:59:08.015648 11307 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56630 (request call id 1324) took 1640 ms (client timeout 2157 ms). Trace:
I20260321 02:59:08.015652 11230 raft_consensus.cc:2468] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:08.005367 12749 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:08.014168 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:08.016198 11307 rpcz_store.cc:269] 0321 02:59:06.363732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:06.487780 (+124048us) service_pool.cc:225] Handling call
0321 02:59:08.003646 (+1515866us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b5400338bd4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:08.003865 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":880477568}
I20260321 02:59:08.032680 12818 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:08.033043 11384 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40934 (request call id 1313) took 1031 ms. Trace:
I20260321 02:59:08.033953 11306 raft_consensus.cc:2368] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:59:08.033496 11384 rpcz_store.cc:276] 0321 02:59:07.001997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:07.002210 (+ 213us) service_pool.cc:225] Handling call
0321 02:59:08.032601 (+1030391us) spinlock_profiling.cc:243] Waited 72.6 ms on lock 0x7b5400112954. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:08.032844 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":166932352}
I20260321 02:59:08.035436 11379 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:08.048830 11310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1315) took 1694 ms. Trace:
I20260321 02:59:08.049541 11310 rpcz_store.cc:276] 0321 02:59:06.354270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:06.354457 (+ 187us) service_pool.cc:225] Handling call
0321 02:59:07.583175 (+1228718us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:07.583340 (+ 165us) raft_consensus.cc:1534] Early marking committed up to index 3
0321 02:59:07.604563 (+ 21223us) log.cc:844] Serialized 14 byte log entry
0321 02:59:07.725146 (+120583us) log.cc:844] Serialized 14 byte log entry
0321 02:59:07.725298 (+ 152us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:07.725340 (+ 42us) raft_consensus.cc:1645] Marking committed up to 3
0321 02:59:07.725380 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:07.725448 (+ 68us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:08.048281 (+322833us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54003386d0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54bc8d 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:08.048517 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":108923,"prepare.queue_time_us":29772,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":71,"raft.queue_time_us":142235,"spinlock_wait_cycles":2860639232,"thread_start_us":11084,"threads_started":1,"wal-append.queue_time_us":11539}
I20260321 02:59:08.052042 11305 raft_consensus.cc:2368] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:59:08.053294 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.054811 11306 raft_consensus.cc:2393] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 2: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:08.056371 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.057756 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.058995 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:07.724727 11304 rpcz_store.cc:276] 0321 02:59:06.413478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:07.455856 (+1042378us) service_pool.cc:225] Handling call
0321 02:59:07.456203 (+ 347us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:07.456407 (+ 204us) raft_consensus.cc:1534] Early marking committed up to index 3
0321 02:59:07.456688 (+ 281us) log.cc:844] Serialized 14 byte log entry
0321 02:59:07.723939 (+267251us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:07.724021 (+ 82us) raft_consensus.cc:1645] Marking committed up to 3
0321 02:59:07.724064 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:07.724123 (+ 59us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:07.724248 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":224996,"prepare.queue_time_us":117,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":100,"raft.queue_time_us":226900,"thread_start_us":41914,"threads_started":1,"wal-append.queue_time_us":46675}
I20260321 02:59:08.059563 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:08.061826 11349 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:08.061234 11304 raft_consensus.cc:1240] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:08.059008 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.059913 11308 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:59:08.062000 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.060534 11235 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:08.073933 11229 raft_consensus.cc:1240] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:08.075224 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:08.075520 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:08.077374 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:08.081379 11227 raft_consensus.cc:1240] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260321 02:59:08.088687 11302 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
W20260321 02:59:08.089706 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 427046us
I20260321 02:59:08.082952 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:08.095521 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.096686 11310 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:59:08.098676 12998 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:08.100018 12998 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.117298 11231 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58214 (request call id 1038) took 1570 ms. Trace:
I20260321 02:59:08.117949 11231 rpcz_store.cc:276] 0321 02:59:06.547295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:06.732286 (+184991us) service_pool.cc:225] Handling call
0321 02:59:06.752016 (+ 19730us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:07.134347 (+382331us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:08.116517 (+982170us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b5400332cd4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:08.117040 (+ 523us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":820039552}
W20260321 02:59:08.120831 11232 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58200 (request call id 1417) took 1993 ms (client timeout 2042 ms). Trace:
W20260321 02:59:08.121443 11232 rpcz_store.cc:269] 0321 02:59:06.127408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:06.127822 (+ 414us) service_pool.cc:225] Handling call
0321 02:59:06.995733 (+867911us) spinlock_profiling.cc:243] Waited 5.25 ms on lock 0x7b5800111398. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b150e760 00007fa8b150e044 00007fa8aaa8f834 00007fa8aaa8f7c2
0321 02:59:08.120328 (+1124595us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x7b5400289bd4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:08.120580 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":988636416}
I20260321 02:59:08.122781 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.123602 11302 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:59:08.126309 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.068621 12749 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.137976 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:08.139348 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:08.140631 11305 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:08.076401 12848 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.144340 11235 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:08.147027 12968 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:08.147462 12961 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:08.147928 12968 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:08.150440 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.145458 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:08.148135 12961 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:08.151888 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.142750 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:08.158236 11236 raft_consensus.cc:2393] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:08.144389 12998 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:08.160085 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:08.163018 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:08.163482 11302 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:08.164471 11383 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:08.164855 12940 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
W20260321 02:59:08.165705 11349 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:08.165573 11977 raft_consensus.cc:1275] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260321 02:59:08.151614 12720 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.169205 11379 raft_consensus.cc:1275] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260321 02:59:08.171752 12880 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260321 02:59:08.172612 12896 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:08.165580 12940 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.173242 12896 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.174989 12940 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:08.171802 11304 raft_consensus.cc:2468] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:59:08.176054 11308 raft_consensus.cc:2393] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 4: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:08.176699 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:08.176954 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.177773 11304 raft_consensus.cc:2393] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 2: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:08.178092 11235 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:08.155552 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:08.182296 11232 raft_consensus.cc:2368] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4.
W20260321 02:59:08.184298 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:08.177352 12720 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:08.188148 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:08.195138 11304 raft_consensus.cc:1275] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:08.181892 12940 consensus_queue.cc:1048] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
W20260321 02:59:08.197110 12783 consensus_meta.cc:287] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.533s user 0.014s sys 0.005s
I20260321 02:59:08.205363 12953 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.206471 11302 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:08.191701 11379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1301) took 3203 ms. Trace:
I20260321 02:59:08.207408 11379 rpcz_store.cc:276] 0321 02:59:04.981201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:05.126325 (+145124us) service_pool.cc:225] Handling call
0321 02:59:05.126938 (+ 613us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:07.403643 (+2276705us) spinlock_profiling.cc:243] Waited 713 ms on lock 0x7b18002bd8e0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac587a26 00007fa8ac4c196e 00007fa8ac54337e 00007fa8ac544bbe 00007fa8ac54ef29 00007fa8ac54f7f6 00007fa8ac54c16f 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34
0321 02:59:08.034909 (+631266us) spinlock_profiling.cc:243] Waited 628 ms on lock 0x7b58000a34d0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac4a848a 00007fa8ac52a827 00007fa8ac5433c7 00007fa8ac544bbe 00007fa8ac54ef29 00007fa8ac54f7f6 00007fa8ac54c16f 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34
0321 02:59:08.169636 (+134727us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:08.184217 (+ 14581us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3085703168}
I20260321 02:59:08.208165 12968 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:08.209417 12848 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:08.208994 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:08.211012 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:08.211509 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:08.212229 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:08.213109 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:08.213308 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:08.214466 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:08.214763 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:08.215224 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:08.216612 11310 raft_consensus.cc:2393] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 4: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:08.218716 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:08.220708 12749 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:08.221618 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:08.221724 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.216647 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:08.224615 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:08.225203 13009 raft_consensus.cc:3055] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:08.225625 13009 raft_consensus.cc:740] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:08.230454 11379 raft_consensus.cc:2393] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 3: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:08.235152 11973 raft_consensus.cc:1275] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260321 02:59:08.237097 13006 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:08.237133 11376 raft_consensus.cc:1275] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260321 02:59:08.242815 12968 consensus_queue.cc:1059] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:08.243387 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:08.244015 12968 raft_consensus.cc:3055] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:59:08.244679 11376 raft_consensus.cc:1240] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260321 02:59:08.244874 12968 raft_consensus.cc:740] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:08.246004 12968 consensus_queue.cc:260] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.247407 12968 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:05.175058 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:08.244143 11306 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:08.223865 11233 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:59:08.207578 12953 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:08.256634 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:08.257045 13056 log.cc:927] Time spent T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.094s user 0.001s sys 0.000s
I20260321 02:59:08.257431 11303 raft_consensus.cc:2468] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:59:08.259222 11381 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:08.261451 12934 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:08.260736 13009 consensus_queue.cc:260] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:08.259143 12749 consensus_queue.cc:1048] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260321 02:59:08.259351 12961 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:08.272998 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:08.274806 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:08.276355 12924 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:08.278123 13053 log.cc:927] Time spent T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.139s user 0.001s sys 0.000s
I20260321 02:59:08.285331 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:08.299515 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:08.300580 11235 raft_consensus.cc:2393] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 3: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:08.303431 11307 raft_consensus.cc:2393] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 3: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:08.301338 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:08.302356 13009 raft_consensus.cc:3060] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:08.317890 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:08.319324 11303 raft_consensus.cc:2393] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 3: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:08.305976 11380 raft_consensus.cc:3060] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:08.327247 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:08.327742 11377 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:59:08.327266 11309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1321) took 1970 ms. Trace:
W20260321 02:59:08.337178 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 97288.2us
I20260321 02:59:08.329895 11309 rpcz_store.cc:276] 0321 02:59:06.356683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:07.182367 (+825684us) service_pool.cc:225] Handling call
0321 02:59:07.211244 (+ 28877us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:07.703548 (+492304us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:07.703609 (+ 61us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:07.704350 (+ 741us) log.cc:844] Serialized 26 byte log entry
0321 02:59:08.093663 (+389313us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:59:08.125376 (+ 31713us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:08.134207 (+ 8831us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:08.321559 (+187352us) raft_consensus.cc:1685] finished
0321 02:59:08.321587 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:08.327104 (+ 5517us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":491435,"raft.queue_time_us":491987,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":112,"spinlock_wait_cycles":1739008,"thread_start_us":304477,"threads_started":1,"wal-append.queue_time_us":324677}
I20260321 02:59:08.339210 11235 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:08.340029 12925 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:08.340232 12896 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:08.340708 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:08.340667 12925 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:08.342401 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:08.343184 12925 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:08.345367 11132 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42530 (request call id 216) took 5011 ms. Trace:
I20260321 02:59:08.345829 11132 rpcz_store.cc:276] 0321 02:59:03.333539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:03.333712 (+ 173us) service_pool.cc:225] Handling call
0321 02:59:03.454376 (+120664us) spinlock_profiling.cc:243] Waited 80.8 ms on lock 0x7b54002d69d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b6470 00007fa8ac55531a 00007fa8b22e365c 00007fa8b23dd5de 00007fa8b08d64a4 00007fa8b08d6432 00007fa8b08d6361 00007fa8b08d62dc 00007fa8b08d5592 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:08.345093 (+4890717us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2582047,"reported_tablets":2,"spinlock_wait_cycles":185906944}
I20260321 02:59:08.347738 11382 raft_consensus.cc:2393] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 5: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:08.355020 11235 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:08.360458 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:08.356794 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:08.364930 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 52708.1us
I20260321 02:59:08.402767 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:08.404110 11379 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:08.405303 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.411274 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:08.408140 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:08.425665 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:08.429225 11349 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:08.435535 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:08.442459 12908 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:08.443300 12908 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:08.445202 11940 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:08.446195 12908 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:08.439256 11377 raft_consensus.cc:2376] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5. Re-sending same reply.
I20260321 02:59:08.450415 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:08.455083 11382 raft_consensus.cc:2393] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 5: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:08.457103 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:08.458762 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:08.459465 11303 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:08.462333 12987 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:08.462971 12987 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.464354 12896 raft_consensus.cc:2749] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:08.468322 11383 raft_consensus.cc:2393] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 5: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
W20260321 02:59:08.459930 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 113727us
I20260321 02:59:08.455776 12924 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:08.476713 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:08.488610 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:08.498144 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:08.501596 11379 raft_consensus.cc:2393] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 3: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
W20260321 02:59:08.504705 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:08.519007 11377 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:08.519929 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:08.525435 11231 raft_consensus.cc:1240] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260321 02:59:08.528520 13011 consensus_queue.cc:1048] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260321 02:59:08.532055 11311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1341) took 1673 ms. Trace:
W20260321 02:59:08.532434 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 116615us
I20260321 02:59:08.532559 11311 rpcz_store.cc:276] 0321 02:59:06.753213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:07.593083 (+839870us) service_pool.cc:225] Handling call
0321 02:59:07.593538 (+ 455us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:07.594013 (+ 475us) raft_consensus.cc:1534] Early marking committed up to index 3
0321 02:59:07.621951 (+ 27938us) log.cc:844] Serialized 14 byte log entry
0321 02:59:08.229290 (+607339us) spinlock_profiling.cc:243] Waited 1.4 ms on lock 0x7b18003763c0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b2300441 00007fa8b230cca1 00007fa8a9498d17 00007fa8b2276b19 00007fa8a94b546b 00007fa8a94b370f 00007fa8a94b59d2 00007fa8ac0e55ab 00007fa8ac0eb0c3 00007fa8ac0eb5da 00007fa8ac556e0a
0321 02:59:08.362281 (+132991us) log.cc:844] Serialized 14 byte log entry
0321 02:59:08.397934 (+ 35653us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:08.406548 (+ 8614us) raft_consensus.cc:1645] Marking committed up to 3
0321 02:59:08.414780 (+ 8232us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:08.418640 (+ 3860us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:08.426754 (+ 8114us) inbound_call.cc:173] Queueing success response
0321 02:59:08.472787 (+ 46033us) spinlock_profiling.cc:243] Waited 2.57 ms on lock 0x7b18003763c0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b2300441 00007fa8b230cca1 00007fa8a94b61c3 00007fa8a94b8591 00007fa8a94b8549 00007fa8a94b84d9 00007fa8a94b84a1 00007fa8a94b779d 0000000000379254 0000000000379099 00007fa8a949974a
Metrics: {"mutex_wait_us":22973,"prepare.queue_time_us":39772,"prepare.run_cpu_time_us":130,"prepare.run_wall_time_us":2741,"raft.queue_time_us":17280,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":27913,"spinlock_wait_cycles":37117440,"thread_start_us":65556,"threads_started":1,"wal-append.queue_time_us":818261}
I20260321 02:59:08.534562 11349 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:08.535717 11349 leader_election.cc:400] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4.
I20260321 02:59:08.540232 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:08.540545 11229 raft_consensus.cc:1275] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260321 02:59:08.515621 11133 catalog_manager.cc:5671] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:59:08.542435 11236 raft_consensus.cc:1240] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:08.544351 12897 raft_consensus.cc:2804] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:08.550010 11349 leader_election.cc:403] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260321 02:59:08.550211 11309 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:08.552212 11200 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:08.552871 11303 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:08.559183 11305 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:08.560225 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:08.561692 11303 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:08.570104 12749 raft_consensus.cc:2804] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:08.572710 12897 raft_consensus.cc:697] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 6 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:08.574678 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.575793 11306 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
W20260321 02:59:08.575736 11275 connection.cc:232] RPC call timeout handler was delayed by -1.1189089329999433s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1503s for a response
I20260321 02:59:08.572885 13016 raft_consensus.cc:2749] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4.
W20260321 02:59:08.580741 13060 log.cc:927] Time spent T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.169s user 0.000s sys 0.001s
W20260321 02:59:08.580992 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 52843.5us
I20260321 02:59:08.586292 11200 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:08.587213 12846 raft_consensus.cc:2749] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:08.587702 11200 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:08.573796 12749 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:08.593724 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:08.594640 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 59188.2us
I20260321 02:59:08.584679 12880 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:08.591943 12846 raft_consensus.cc:2749] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:08.593140 11133 catalog_manager.cc:5671] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:59:08.592926 12749 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:08.603931 11275 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:08.604240 11306 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 5
I20260321 02:59:08.593164 11380 raft_consensus.cc:2468] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:08.599545 12987 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:08.611424 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.615167 11305 raft_consensus.cc:2393] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 3: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
W20260321 02:59:08.616309 11349 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:08.621351 11303 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:08.597381 12897 consensus_queue.cc:237] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.596108 12880 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.625778 11200 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:08.627022 12880 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:08.629792 11942 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:08.632999 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:08.633898 11303 raft_consensus.cc:2393] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 3: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:08.636472 11377 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:08.632907 11133 catalog_manager.cc:5671] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:59:08.622406 13009 raft_consensus.cc:2749] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:08.641152 11130 catalog_manager.cc:5671] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 1 to 2, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 2 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
W20260321 02:59:08.643244 13063 log.cc:927] Time spent T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.181s user 0.004s sys 0.000s
I20260321 02:59:08.644272 12924 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:08.649688 11200 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:08.650460 11309 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:08.639453 11134 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42502 (request call id 223) took 6469 ms. Trace:
I20260321 02:59:08.647096 11133 catalog_manager.cc:5671] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260321 02:59:08.651017 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 61092.5us
I20260321 02:59:08.649603 12924 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.652904 12846 raft_consensus.cc:2804] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:08.653558 12846 raft_consensus.cc:697] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:08.654904 12924 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:08.662467 11381 raft_consensus.cc:2410] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260321 02:59:08.661206 11200 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:08.665457 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:08.666263 11309 raft_consensus.cc:2393] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 5: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:08.654533 12846 consensus_queue.cc:237] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.657176 13011 raft_consensus.cc:2804] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:59:08.670162 12856 raft_consensus.cc:2804] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:08.673071 12856 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:08.673444 12856 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:08.660640 12997 consensus_queue.cc:1059] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260321 02:59:08.660065 12953 raft_consensus.cc:2804] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election won for term 5
W20260321 02:59:08.677064 13062 log.cc:927] Time spent T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.103s user 0.000s sys 0.001s
I20260321 02:59:08.677737 11275 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:08.683727 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.683321 12848 raft_consensus.cc:3055] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:08.685190 11130 catalog_manager.cc:5671] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 1 to 3, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 3 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:08.679517 13011 raft_consensus.cc:697] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:08.686430 12848 raft_consensus.cc:740] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:08.695132 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.684749 12953 raft_consensus.cc:697] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:08.698050 11306 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:08.688273 12924 raft_consensus.cc:2749] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:08.702184 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.702420 12848 consensus_queue.cc:260] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.703902 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.706017 11306 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:08.706089 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:08.706983 11303 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:08.708359 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.709456 11309 raft_consensus.cc:2393] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 5: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:08.710295 12953 consensus_queue.cc:237] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.710004 11200 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:08.714232 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 46733.8us
I20260321 02:59:08.719025 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:08.704986 12848 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:08.722363 12856 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.723882 13011 consensus_queue.cc:237] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:08.730022 11349 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:08.733559 12968 raft_consensus.cc:2749] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260321 02:59:08.744777 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:08.719115 12749 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:08.747066 11303 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:08.748464 12997 raft_consensus.cc:2804] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:08.749096 12997 raft_consensus.cc:697] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:08.750140 12997 consensus_queue.cc:237] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.751446 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:08.752058 12749 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:08.651230 11134 rpcz_store.cc:276] 0321 02:59:02.169509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:02.169709 (+ 200us) service_pool.cc:225] Handling call
0321 02:59:08.639215 (+6469506us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:59:02.855285 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:59:03.079817 (+224532us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:59:03.081534 (+ 1717us) write_op.cc:435] Acquired schema lock
0321 02:59:03.084482 (+ 2948us) tablet.cc:661] Decoding operations
0321 02:59:03.207900 (+123418us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:59:03.208019 (+ 119us) write_op.cc:641] Partition lock acquired for write op
0321 02:59:03.208069 (+ 50us) tablet.cc:684] Acquiring locks for 4 operations
0321 02:59:03.208648 (+ 579us) tablet.cc:700] Row locks acquired
0321 02:59:03.208676 (+ 28us) write_op.cc:260] PREPARE: finished
0321 02:59:03.209011 (+ 335us) write_op.cc:270] Start()
0321 02:59:03.209168 (+ 157us) write_op.cc:276] Timestamp: P: 1774061943208941 usec, L: 0
0321 02:59:03.209200 (+ 32us) op_driver.cc:348] REPLICATION: starting
0321 02:59:03.210209 (+ 1009us) log.cc:844] Serialized 1682 byte log entry
0321 02:59:03.612321 (+402112us) op_driver.cc:464] REPLICATION: finished
0321 02:59:04.026616 (+414295us) write_op.cc:301] APPLY: starting
0321 02:59:04.026719 (+ 103us) tablet.cc:1366] starting BulkCheckPresence
0321 02:59:04.026886 (+ 167us) tablet.cc:1369] finished BulkCheckPresence
0321 02:59:04.108009 (+ 81123us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:59:04.283420 (+175411us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:59:04.309862 (+ 26442us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0321 02:59:04.319044 (+ 9182us) write_op.cc:312] APPLY: finished
0321 02:59:04.385558 (+ 66514us) log.cc:844] Serialized 41 byte log entry
0321 02:59:04.403164 (+ 17606us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:04.422938 (+ 19774us) write_op.cc:454] Released schema lock
0321 02:59:04.422985 (+ 47us) write_op.cc:341] FINISH: Updating metrics
0321 02:59:04.423342 (+ 357us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:04.423375 (+ 33us) write_op.cc:454] Released schema lock
0321 02:59:05.207276 (+783901us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b18002faca0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b2300441 00007fa8b230cca1 00007fa8a94b62f2 00007fa8a94b8591 00007fa8a94b8549 00007fa8a94b84d9 00007fa8a94b84a1 00007fa8a94b779d 0000000000379254 0000000000379099 00007fa8a949974a
Metrics: {"mutex_wait_us":717237,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":378676,"apply.run_cpu_time_us":4485,"apply.run_wall_time_us":396737,"num_ops":4,"prepare.queue_time_us":315783,"prepare.run_cpu_time_us":21067,"prepare.run_wall_time_us":705847,"raft.queue_time_us":163619,"raft.run_cpu_time_us":18073,"raft.run_wall_time_us":380800,"replication_time_us":389477,"spinlock_wait_cycles":39672192,"thread_start_us":638768,"threads_started":4,"wal-append.queue_time_us":222660,"wal-append.run_cpu_time_us":13862,"wal-append.run_wall_time_us":2033953}]]}
I20260321 02:59:08.757774 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:08.766165 13070 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:08.766873 13070 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:08.780202 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:08.814924 11275 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:08.814644 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 73943.1us
I20260321 02:59:08.820967 12720 consensus_queue.cc:1048] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260321 02:59:08.824682 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 123793us
I20260321 02:59:08.826939 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.827550 11306 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:08.833905 12924 raft_consensus.cc:2749] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260321 02:59:08.839028 11942 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 2.070s (SENT) [suppressed 20 similar messages]
I20260321 02:59:08.851208 13070 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:08.852114 12924 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:08.852799 12924 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.855165 12924 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:08.864192 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 131295us
I20260321 02:59:08.870985 11349 leader_election.cc:400] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267) with higher term. Message: Invalid argument: T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:59:08.871788 11349 leader_election.cc:403] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260321 02:59:08.872493 12968 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 3
W20260321 02:59:08.878448 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 57006.2us
I20260321 02:59:08.886143 11200 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:08.884009 12856 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:08.885596 12897 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:08.890120 12953 raft_consensus.cc:2804] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:08.890362 12897 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.890787 12953 raft_consensus.cc:697] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:08.892410 11977 raft_consensus.cc:1240] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:08.891970 12953 consensus_queue.cc:237] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:08.892886 12897 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:08.893838 11980 raft_consensus.cc:1217] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[3.5-3.5] Dedup: 3.5->[]
I20260321 02:59:08.894322 11980 raft_consensus.cc:1240] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260321 02:59:08.897037 11275 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:08.897948 12924 raft_consensus.cc:2804] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:08.898345 12924 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:08.898713 12924 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:08.899259 11275 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:08.903112 13009 raft_consensus.cc:2804] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:08.903569 13009 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:08.904003 13009 raft_consensus.cc:3060] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:08.903774 12990 consensus_queue.cc:1059] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:08.904896 12908 raft_consensus.cc:3055] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:08.905359 12908 raft_consensus.cc:740] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:08.906432 12908 consensus_queue.cc:260] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.907281 12968 raft_consensus.cc:2749] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267) with higher term. Message: Invalid argument: T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:59:08.910063 11349 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:08.912364 12908 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:08.915721 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.915864 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.916466 11305 raft_consensus.cc:2468] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:08.912905 11200 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:08.918430 12856 raft_consensus.cc:2764] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260321 02:59:08.913604 12968 raft_consensus.cc:2804] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:08.919427 12968 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:08.919849 12968 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:08.917944 11200 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:08.928241 12856 raft_consensus.cc:2804] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:08.928718 12856 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:08.929113 12856 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:08.911880 11349 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:08.915369 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:08.937323 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:08.939249 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.936264 12897 raft_consensus.cc:2749] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:08.957590 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:08.959106 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 47339us
I20260321 02:59:08.939612 12934 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:08.967378 11349 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:08.976953 12880 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:08.980955 11349 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:08.983538 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.979182 13009 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:08.985030 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:08.979139 12896 raft_consensus.cc:2764] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260321 02:59:08.986668 11305 raft_consensus.cc:1240] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 2. Current term is 3. Ops: []
I20260321 02:59:08.977456 12897 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:08.982185 13083 log.cc:927] Time spent T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260321 02:59:08.983635 12880 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.996628 12897 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:08.997164 12924 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.033300 12924 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:08.983574 12934 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:09.034293 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:09.029301 12880 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:09.036511 11303 raft_consensus.cc:3060] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:09.039443 12934 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:09.004417 12990 consensus_queue.cc:1059] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 5, Time since last communication: 0.001s
I20260321 02:59:09.040675 12990 raft_consensus.cc:3055] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:09.041074 12990 raft_consensus.cc:740] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:09.040901 12968 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.005323 12896 raft_consensus.cc:2804] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:09.005851 11349 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:09.012741 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.042541 12896 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:09.043018 12896 raft_consensus.cc:3060] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:09.014125 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.014904 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.044433 11349 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:09.045280 12987 raft_consensus.cc:2749] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:09.046008 11349 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:09.046917 12987 raft_consensus.cc:2749] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:09.047380 11349 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:09.048532 12987 raft_consensus.cc:2804] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:09.048946 12987 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:09.049158 11349 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:09.050204 12880 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:09.025207 12856 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.041926 12990 consensus_queue.cc:260] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:09.053031 12856 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:09.053787 12990 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:09.054214 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:09.033851 12897 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:09.043656 12961 raft_consensus.cc:2804] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:09.056164 12961 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:09.057087 12961 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
W20260321 02:59:09.057567 12924 consensus_queue.cc:1175] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" responder_term: 4 status { last_received { term: 1 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260321 02:59:09.014910 11275 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.049386 12987 raft_consensus.cc:3060] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:09.018543 13009 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:09.061066 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 46276.2us
I20260321 02:59:09.061306 13009 raft_consensus.cc:2749] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:09.061839 11275 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.063521 11275 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.064021 11381 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:09.064303 13009 raft_consensus.cc:2749] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:09.065197 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:09.065846 12968 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:09.066565 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:09.070575 11275 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.078737 12908 raft_consensus.cc:2749] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:09.079356 11275 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.071195 12924 raft_consensus.cc:2749] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:09.074465 12987 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:09.080611 11275 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:09.076326 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 80519.9us
I20260321 02:59:09.081579 12987 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:09.081903 11275 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.082688 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:09.074522 12880 raft_consensus.cc:2749] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:09.084192 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:09.084664 13009 raft_consensus.cc:2749] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:09.082686 12924 raft_consensus.cc:2749] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:09.080615 12908 raft_consensus.cc:2749] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260321 02:59:09.082747 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 112495us
I20260321 02:59:09.087237 11200 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:09.087867 13086 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:09.088158 11200 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:09.088419 13086 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.089599 12749 raft_consensus.cc:2804] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:59:09.076870 11303 raft_consensus.cc:2468] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:09.090130 12749 raft_consensus.cc:697] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:09.090533 13086 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:09.091388 13084 raft_consensus.cc:2804] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:09.091131 12749 consensus_queue.cc:237] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.092216 13084 raft_consensus.cc:697] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:09.088431 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:09.093201 13084 consensus_queue.cc:237] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.094117 11349 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:09.077373 12961 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:09.097002 11309 raft_consensus.cc:1275] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260321 02:59:09.099754 11309 pending_rounds.cc:85] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da: Aborting all ops after (but not including) 4
I20260321 02:59:09.098373 11303 raft_consensus.cc:1275] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:09.098709 11380 raft_consensus.cc:1275] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260321 02:59:09.098814 12990 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:09.100035 11309 pending_rounds.cc:107] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260321 02:59:09.102924 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:09.104111 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:09.104452 12976 consensus_queue.cc:1048] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260321 02:59:09.105376 12880 raft_consensus.cc:2804] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:59:09.105788 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:09.109007 11311 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:09.109371 11309 raft_consensus.cc:2889] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:09.109634 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:09.115190 11383 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:09.117089 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:09.117970 11349 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:09.118207 11303 raft_consensus.cc:2376] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5. Re-sending same reply.
W20260321 02:59:09.120198 11200 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:09.120944 11376 raft_consensus.cc:1240] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 2. Current term is 3. Ops: []
I20260321 02:59:09.123536 11377 raft_consensus.cc:1275] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:59:09.131076 11200 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.132728 12846 consensus_queue.cc:1059] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:59:09.133708 13076 raft_consensus.cc:2764] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260321 02:59:09.133940 12846 raft_consensus.cc:3055] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:09.134366 12846 raft_consensus.cc:740] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:09.133677 11200 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.135237 12997 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:09.135874 12997 raft_consensus.cc:697] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:09.124791 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:09.137123 12997 consensus_queue.cc:237] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.095504 12896 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.145265 13076 consensus_queue.cc:1059] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:59:09.146378 12736 raft_consensus.cc:3055] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:59:09.147195 11200 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.147351 12736 raft_consensus.cc:740] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:09.148737 12736 consensus_queue.cc:260] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.149737 13076 raft_consensus.cc:2804] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:09.150189 13076 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:09.150520 13076 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:09.150753 12736 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:09.126116 11275 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:09.157713 11311 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:09.161965 11383 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:09.168211 11382 raft_consensus.cc:1240] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 2. Current term is 3. Ops: []
W20260321 02:59:09.169363 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 63065.9us
I20260321 02:59:09.170326 11349 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:09.135375 12846 consensus_queue.cc:260] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.161562 12990 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.185132 12990 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:09.193418 12961 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:09.168160 12968 raft_consensus.cc:2749] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:09.176900 12908 raft_consensus.cc:2804] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:59:09.176700 12896 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:09.204310 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:09.225406 12908 raft_consensus.cc:697] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:09.239388 12908 consensus_queue.cc:237] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.226773 12987 raft_consensus.cc:2749] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:09.228018 12880 raft_consensus.cc:697] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 2 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:09.253556 12925 consensus_queue.cc:1059] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260321 02:59:09.271191 12880 consensus_queue.cc:237] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.272692 11980 raft_consensus.cc:1275] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260321 02:59:09.276542 12990 consensus_queue.cc:1059] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:09.277949 12990 raft_consensus.cc:3055] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:09.278430 12990 raft_consensus.cc:740] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:09.280157 13009 consensus_queue.cc:1048] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:59:09.283414 12976 raft_consensus.cc:3055] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:59:09.283901 12976 raft_consensus.cc:740] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:09.284842 12976 consensus_queue.cc:260] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:09.286145 12976 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:09.289614 12846 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:09.291604 11384 raft_consensus.cc:1240] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 2. Current term is 3. Ops: []
W20260321 02:59:09.261010 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 107351us
I20260321 02:59:09.279420 12990 consensus_queue.cc:260] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.275564 13076 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.311695 11133 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44688 (request call id 128) took 1430 ms. Trace:
I20260321 02:59:09.338382 11349 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:09.341624 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 135691us
I20260321 02:59:09.357331 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.359285 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.360131 11305 raft_consensus.cc:2376] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5. Re-sending same reply.
I20260321 02:59:09.361495 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.362018 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.362748 11302 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:09.364321 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:09.330359 12908 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:09.364548 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:09.365164 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.352553 12848 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:09.366880 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.362087 11305 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:59:09.369031 11230 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:09.369351 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.369889 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:09.369956 11303 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:09.368948 11275 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.370723 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:09.371161 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.372483 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:09.358115 13009 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:09.373864 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.374337 11304 raft_consensus.cc:1275] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260321 02:59:09.387030 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.387097 12908 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.396162 12990 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:09.396467 11942 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:09.397125 12908 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:09.397940 11307 raft_consensus.cc:2393] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 4: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:09.398569 11942 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.398963 13009 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.399681 13011 raft_consensus.cc:2764] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260321 02:59:09.336314 11133 rpcz_store.cc:276] 0321 02:59:07.882044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:07.883506 (+ 1462us) service_pool.cc:225] Handling call
0321 02:59:09.311476 (+1427970us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:59:08.856990 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:59:08.857804 (+ 814us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:59:08.857862 (+ 58us) write_op.cc:435] Acquired schema lock
0321 02:59:08.857882 (+ 20us) tablet.cc:661] Decoding operations
0321 02:59:08.858581 (+ 699us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:59:08.858674 (+ 93us) write_op.cc:641] Partition lock acquired for write op
0321 02:59:08.858706 (+ 32us) tablet.cc:684] Acquiring locks for 4 operations
0321 02:59:08.859219 (+ 513us) tablet.cc:700] Row locks acquired
0321 02:59:08.859237 (+ 18us) write_op.cc:260] PREPARE: finished
0321 02:59:08.859392 (+ 155us) write_op.cc:270] Start()
0321 02:59:08.859538 (+ 146us) write_op.cc:276] Timestamp: P: 1774061948859326 usec, L: 0
0321 02:59:08.859574 (+ 36us) op_driver.cc:348] REPLICATION: starting
0321 02:59:08.887928 (+ 28354us) log.cc:844] Serialized 1662 byte log entry
0321 02:59:09.085384 (+197456us) op_driver.cc:464] REPLICATION: finished
0321 02:59:09.096750 (+ 11366us) write_op.cc:301] APPLY: starting
0321 02:59:09.096865 (+ 115us) tablet.cc:1366] starting BulkCheckPresence
0321 02:59:09.097016 (+ 151us) tablet.cc:1369] finished BulkCheckPresence
0321 02:59:09.097340 (+ 324us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:59:09.098318 (+ 978us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:59:09.098615 (+ 297us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0321 02:59:09.098675 (+ 60us) write_op.cc:312] APPLY: finished
0321 02:59:09.099007 (+ 332us) log.cc:844] Serialized 41 byte log entry
0321 02:59:09.126768 (+ 27761us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:09.160248 (+ 33480us) write_op.cc:454] Released schema lock
0321 02:59:09.162315 (+ 2067us) write_op.cc:341] FINISH: Updating metrics
0321 02:59:09.193369 (+ 31054us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:09.193406 (+ 37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":564026,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":11068,"apply.run_cpu_time_us":3549,"apply.run_wall_time_us":96613,"num_ops":4,"prepare.queue_time_us":154918,"prepare.run_cpu_time_us":21997,"prepare.run_wall_time_us":135990,"raft.queue_time_us":101725,"raft.run_cpu_time_us":33339,"raft.run_wall_time_us":93214,"replication_time_us":225631,"thread_start_us":248294,"threads_started":4,"wal-append.queue_time_us":85519}]]}
I20260321 02:59:09.372560 11305 raft_consensus.cc:2393] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 3: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:09.387921 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.388466 11235 raft_consensus.cc:1275] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260321 02:59:09.421973 11311 raft_consensus.cc:2376] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6. Re-sending same reply.
I20260321 02:59:09.388351 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.423147 11306 raft_consensus.cc:2410] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260321 02:59:09.343614 13097 raft_consensus.cc:2764] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260321 02:59:09.373368 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:09.390117 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:09.391601 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:09.404690 12925 consensus_queue.cc:1059] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260321 02:59:09.392844 11275 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.392985 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:09.427964 12925 raft_consensus.cc:3055] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:59:09.394131 13086 raft_consensus.cc:2749] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:09.428406 12925 raft_consensus.cc:740] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:09.400223 12976 raft_consensus.cc:2749] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:09.401106 13009 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:09.387530 11308 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:59:09.403753 11310 raft_consensus.cc:2368] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:59:09.405690 11942 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.410465 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:09.410118 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:09.409233 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:09.433579 11233 raft_consensus.cc:2393] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 3: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
I20260321 02:59:09.434782 11275 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.420091 13076 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:09.421816 11130 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42514 (request call id 209) took 7119 ms. Trace:
I20260321 02:59:09.409759 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:09.439479 11942 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.438226 11130 rpcz_store.cc:276] 0321 02:59:02.274942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:02.281581 (+ 6639us) service_pool.cc:225] Handling call
0321 02:59:09.393969 (+7112388us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:59:09.072604 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:59:09.073224 (+ 620us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:59:09.073263 (+ 39us) write_op.cc:435] Acquired schema lock
0321 02:59:09.073280 (+ 17us) tablet.cc:661] Decoding operations
0321 02:59:09.073735 (+ 455us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:59:09.073930 (+ 195us) write_op.cc:641] Partition lock acquired for write op
0321 02:59:09.073970 (+ 40us) tablet.cc:684] Acquiring locks for 2 operations
0321 02:59:09.074359 (+ 389us) tablet.cc:700] Row locks acquired
0321 02:59:09.074389 (+ 30us) write_op.cc:260] PREPARE: finished
0321 02:59:09.074572 (+ 183us) write_op.cc:270] Start()
0321 02:59:09.074727 (+ 155us) write_op.cc:276] Timestamp: P: 1774061949074492 usec, L: 0
0321 02:59:09.074756 (+ 29us) op_driver.cc:348] REPLICATION: starting
0321 02:59:09.077889 (+ 3133us) log.cc:844] Serialized 917 byte log entry
0321 02:59:09.096882 (+ 18993us) op_driver.cc:464] REPLICATION: finished
0321 02:59:09.194201 (+ 97319us) write_op.cc:301] APPLY: starting
0321 02:59:09.194279 (+ 78us) tablet.cc:1366] starting BulkCheckPresence
0321 02:59:09.194387 (+ 108us) tablet.cc:1369] finished BulkCheckPresence
0321 02:59:09.194465 (+ 78us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:59:09.205954 (+ 11489us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:59:09.232758 (+ 26804us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0321 02:59:09.237401 (+ 4643us) write_op.cc:312] APPLY: finished
0321 02:59:09.256778 (+ 19377us) log.cc:844] Serialized 29 byte log entry
0321 02:59:09.256965 (+ 187us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:09.257052 (+ 87us) write_op.cc:454] Released schema lock
0321 02:59:09.257095 (+ 43us) write_op.cc:341] FINISH: Updating metrics
0321 02:59:09.257644 (+ 549us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:09.257666 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":99203,"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":97080,"apply.run_cpu_time_us":2419,"apply.run_wall_time_us":63323,"num_ops":2,"prepare.queue_time_us":333637,"prepare.run_cpu_time_us":3367,"prepare.run_wall_time_us":5652,"replication_time_us":21955,"thread_start_us":70713,"threads_started":1}]]}
I20260321 02:59:09.441715 13011 raft_consensus.cc:2749] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260321 02:59:09.425341 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 54546.2us
I20260321 02:59:09.442427 13011 consensus_queue.cc:1059] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260321 02:59:09.442873 11200 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:09.387193 12848 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.437693 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:09.436059 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:09.469131 11384 raft_consensus.cc:2393] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 4: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:09.470245 11227 raft_consensus.cc:1240] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260321 02:59:09.470405 11378 raft_consensus.cc:1240] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:09.438298 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:09.448709 12848 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:09.474946 11382 raft_consensus.cc:2393] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 3: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:09.474819 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:09.476395 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:09.475940 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:09.481696 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:09.482549 11381 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:09.431341 13086 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:09.483925 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:09.449364 13070 raft_consensus.cc:2749] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:09.481237 11134 catalog_manager.cc:5671] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:59:09.403187 13092 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:09.471975 12924 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:09.490284 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 47547.9us
I20260321 02:59:09.472297 12908 raft_consensus.cc:2749] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:09.489349 11134 catalog_manager.cc:5671] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 4, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 4 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:09.497783 11134 catalog_manager.cc:5671] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:59:09.484599 12848 raft_consensus.cc:2804] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:09.504002 12848 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:09.504472 12848 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
W20260321 02:59:09.505070 13101 log.cc:927] Time spent T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.103s user 0.002s sys 0.000s
I20260321 02:59:09.480621 13097 consensus_queue.cc:1048] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.021s
W20260321 02:59:09.507994 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 73255.4us
I20260321 02:59:09.495085 12925 consensus_queue.cc:260] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:09.512593 11349 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:09.512959 12925 raft_consensus.cc:3060] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:09.513407 13097 raft_consensus.cc:2749] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:09.502002 12924 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.515436 12924 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:09.509888 11134 catalog_manager.cc:5671] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 2, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 2 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:09.502229 12934 raft_consensus.cc:2749] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:09.491988 13076 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:09.511762 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.513343 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:09.523936 11307 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:09.523779 13076 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:09.526124 13076 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:09.502540 11200 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.527262 11349 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:09.527381 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:09.528167 13097 raft_consensus.cc:2804] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:09.528759 13097 raft_consensus.cc:697] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:09.529845 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:09.529678 13097 consensus_queue.cc:237] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.532948 12848 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:09.518970 11134 catalog_manager.cc:5671] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 3 to 4, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 4 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:09.521229 11308 raft_consensus.cc:1275] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260321 02:59:09.494768 13086 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:09.537640 11134 catalog_manager.cc:5671] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 3 to 5, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 5 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:09.522749 11227 raft_consensus.cc:1240] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:09.542233 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:09.543876 11200 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:09.544654 13076 raft_consensus.cc:2749] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:09.547286 11232 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:09.545105 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:09.548558 11230 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:09.547371 11134 catalog_manager.cc:5671] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 3 to 4, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 4 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:09.544669 13086 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:09.550398 11134 catalog_manager.cc:5671] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 2 to 3, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 3 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:09.554551 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:09.561553 11349 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:09.562964 11349 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:09.555811 13086 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:09.564435 12934 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:09.564556 11275 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:09.564620 12968 raft_consensus.cc:2804] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:09.553197 13094 raft_consensus.cc:2749] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:09.565358 12968 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:09.565443 11275 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:09.565073 12934 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.565874 12968 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:09.555882 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:09.566768 12961 raft_consensus.cc:2749] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:09.567595 13108 raft_consensus.cc:2804] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:09.570212 13108 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:09.564705 13086 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.572547 13086 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:09.572386 11349 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:09.573570 12961 raft_consensus.cc:2804] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:09.573975 12961 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:09.574265 12934 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:09.574331 12961 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:09.570785 13108 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:09.576524 12934 consensus_queue.cc:1048] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:59:09.577847 11275 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:09.494390 13092 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.583508 13086 raft_consensus.cc:2749] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:09.575217 11349 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:09.589629 11349 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:09.594213 13093 raft_consensus.cc:2749] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:09.590996 12987 raft_consensus.cc:2804] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:09.609875 12987 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:09.610284 12987 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:09.612326 12968 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.612926 12961 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.615223 11275 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:09.616415 11275 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:09.617802 12934 raft_consensus.cc:2804] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:09.617902 13109 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:09.618275 12934 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:09.618383 13109 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:09.618708 12934 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:09.618786 13109 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:09.624801 13086 consensus_queue.cc:1059] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:09.626643 13086 raft_consensus.cc:3055] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:09.627008 13086 raft_consensus.cc:740] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:09.628432 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:09.628594 11942 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:09.630344 13011 raft_consensus.cc:2764] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260321 02:59:09.627787 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:09.631632 11227 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:09.630903 12925 consensus_queue.cc:1048] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260321 02:59:09.635046 11200 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.615005 12961 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:09.636139 13076 raft_consensus.cc:2749] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:09.638130 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:09.639843 11200 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:09.614979 12968 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:09.627812 13086 consensus_queue.cc:260] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.606170 11349 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:09.621121 12848 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:09.641621 12736 raft_consensus.cc:2749] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:09.648938 12968 raft_consensus.cc:2749] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260321 02:59:09.650074 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 44008us
I20260321 02:59:09.625429 13108 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.644084 13086 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:09.652992 11200 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.644554 13092 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:09.664716 11200 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.666839 12846 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:09.667325 12736 raft_consensus.cc:2749] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:09.679096 13109 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.680053 12934 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.682184 12934 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:09.660112 12961 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:09.666332 11275 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:09.662472 13108 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:09.671211 13092 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:09.676846 12987 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.690965 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:09.691602 11227 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:09.666088 11132 catalog_manager.cc:5671] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:09.690696 13092 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.690994 11200 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.693567 11132 catalog_manager.cc:5671] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:59:09.696120 11132 catalog_manager.cc:5671] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 3 to 4, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 4 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:09.704986 11200 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:09.708858 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 50538.6us
I20260321 02:59:09.710260 11349 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:09.710801 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:09.711377 11230 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:09.711645 11349 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:09.713697 12880 raft_consensus.cc:2749] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:09.713973 13092 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:09.714736 13093 raft_consensus.cc:2804] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:09.715126 13093 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:09.712590 12968 consensus_queue.cc:1048] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.003s
I20260321 02:59:09.715494 13093 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:09.712255 13076 raft_consensus.cc:2804] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:09.716313 13076 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:09.716724 13076 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:09.715485 12720 consensus_queue.cc:1048] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:59:09.717365 12968 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:09.717919 12968 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.720041 13094 consensus_queue.cc:1048] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:59:09.722817 11236 raft_consensus.cc:1275] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:59:09.693445 12961 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.724723 13086 consensus_queue.cc:1048] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:59:09.725175 11200 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:09.725730 12961 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:09.725914 13094 raft_consensus.cc:2804] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:59:09.726465 13094 raft_consensus.cc:697] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:09.712862 11349 leader_election.cc:400] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267) with higher term. Message: Invalid argument: T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3.
I20260321 02:59:09.729171 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:09.729187 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:09.732167 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:09.733006 11233 raft_consensus.cc:2393] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:09.732259 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:09.734041 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:09.734813 11235 raft_consensus.cc:2393] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 3: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:09.708374 12736 raft_consensus.cc:2749] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:09.736214 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:09.727311 13094 consensus_queue.cc:237] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.736822 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:09.738435 11230 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:09.704600 12934 raft_consensus.cc:2804] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:09.730327 11132 catalog_manager.cc:5671] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:09.745353 11275 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:09.747144 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:09.748929 11233 raft_consensus.cc:1275] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260321 02:59:09.727030 12987 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:09.748580 11132 catalog_manager.cc:5671] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:09.752693 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:09.754406 11228 raft_consensus.cc:1275] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260321 02:59:09.755524 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:09.756472 11233 raft_consensus.cc:1240] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 2. Current term is 3. Ops: []
I20260321 02:59:09.749851 12968 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:09.750918 13086 raft_consensus.cc:2804] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:09.765517 13086 raft_consensus.cc:697] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:09.766564 13086 consensus_queue.cc:237] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.750288 12961 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:09.767921 13093 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.771603 13093 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:09.768327 12736 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:09.768409 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 55679.4us
I20260321 02:59:09.773617 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.774451 11307 raft_consensus.cc:2410] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260321 02:59:09.774580 11275 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.775930 11233 raft_consensus.cc:1240] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:09.777900 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:09.768503 13109 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:09.778932 11378 raft_consensus.cc:1240] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 2. Current term is 3. Ops: []
I20260321 02:59:09.769487 13076 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.779526 12736 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:09.780339 11229 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:59:09.781488 12736 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:09.782506 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:09.768700 12961 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.781328 13076 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:09.746781 12934 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:09.762466 11132 catalog_manager.cc:5671] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:09.787988 11275 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.792189 11376 raft_consensus.cc:1240] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:09.794101 11307 raft_consensus.cc:1275] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:59:09.795373 11275 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.797629 11308 raft_consensus.cc:1240] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:09.776238 13009 raft_consensus.cc:2749] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:09.800251 11384 raft_consensus.cc:1275] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:09.786067 12961 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:09.801868 11275 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.804669 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:09.798910 11132 catalog_manager.cc:5671] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:59:09.807358 11275 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.810298 11378 raft_consensus.cc:1275] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:09.783944 11227 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:09.793452 12934 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:09.817670 11377 raft_consensus.cc:1275] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 02:59:09.818537 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:09.824687 11384 raft_consensus.cc:1275] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:09.799484 13109 raft_consensus.cc:2749] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:09.820159 11275 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.821239 11307 raft_consensus.cc:1275] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:09.821311 11308 raft_consensus.cc:1275] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:09.821566 11942 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:09.822728 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:09.826653 11376 raft_consensus.cc:2393] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 5: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:09.827530 11942 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:09.828305 11275 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.829476 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:09.830132 11377 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:09.829869 11381 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:09.829339 13086 raft_consensus.cc:2749] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:09.830005 13009 raft_consensus.cc:2749] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:09.858260 12736 consensus_queue.cc:1048] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:09.847558 12848 consensus_queue.cc:1048] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260321 02:59:09.857195 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.858177 13094 consensus_queue.cc:1059] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260321 02:59:09.861012 11305 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:09.856564 11310 raft_consensus.cc:1275] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:09.851727 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:09.853837 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:09.858036 11303 raft_consensus.cc:1275] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:59:09.832288 11275 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.854511 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:09.819060 12990 raft_consensus.cc:2749] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:09.863762 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:09.855157 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:09.864485 11381 raft_consensus.cc:2393] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 5: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:09.812539 11132 catalog_manager.cc:5671] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 2 to 4, now has a pending config: VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1), VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2), VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 4 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } } }
I20260321 02:59:09.819413 13108 raft_consensus.cc:2749] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:09.846336 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:09.858624 11385 raft_consensus.cc:1275] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:59:09.862228 11377 raft_consensus.cc:2410] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260321 02:59:09.857913 11200 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.873507 11376 raft_consensus.cc:1275] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260321 02:59:09.872406 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:09.873858 11376 pending_rounds.cc:85] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5: Aborting all ops after (but not including) 2
I20260321 02:59:09.873739 11200 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:09.874241 11376 pending_rounds.cc:107] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260321 02:59:09.874436 11382 raft_consensus.cc:2393] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 4: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:09.874578 11376 raft_consensus.cc:2889] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:09.868930 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:09.869943 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:09.872546 11308 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:09.866472 11234 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:09.830464 11380 raft_consensus.cc:1275] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:09.875633 11381 raft_consensus.cc:2393] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 5: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:09.870950 11132 catalog_manager.cc:5671] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 1 to 3, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 3 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:09.856810 11378 raft_consensus.cc:1275] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260321 02:59:09.876451 13099 raft_consensus.cc:2749] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:09.831554 12976 raft_consensus.cc:2749] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:09.831444 12924 raft_consensus.cc:2749] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:09.829640 13107 raft_consensus.cc:2749] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:09.850690 12846 raft_consensus.cc:2749] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:09.867125 13109 raft_consensus.cc:2749] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:09.828729 13093 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:09.912510 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:09.913271 11376 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 5
I20260321 02:59:09.912968 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:09.915053 11381 raft_consensus.cc:2393] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 5: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:09.917832 11349 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:09.831630 13011 consensus_queue.cc:1059] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260321 02:59:09.831233 13070 consensus_queue.cc:1048] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260321 02:59:09.902077 11385 raft_consensus.cc:1275] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:59:09.880260 12736 raft_consensus.cc:2804] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:09.920316 11349 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:09.924654 11349 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:09.911760 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 47068us
I20260321 02:59:09.926545 11349 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:09.927913 11275 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:09.931792 11275 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:09.934707 13097 raft_consensus.cc:2749] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:09.920133 12961 raft_consensus.cc:2749] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:09.899564 13096 consensus_queue.cc:1059] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260321 02:59:09.943274 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:09.951061 11311 raft_consensus.cc:2376] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5. Re-sending same reply.
I20260321 02:59:09.943856 13092 consensus_queue.cc:1048] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:59:09.927067 12968 raft_consensus.cc:2749] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:09.944659 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.958096 11304 raft_consensus.cc:2393] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 3: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:09.944761 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:09.953346 11349 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:09.961629 11309 raft_consensus.cc:2368] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5.
I20260321 02:59:09.959260 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:09.965443 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:09.968869 11275 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.931595 12736 raft_consensus.cc:697] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:09.921643 13093 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:09.971788 13103 consensus_queue.cc:1048] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:09.971778 11200 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:09.971925 12846 consensus_queue.cc:1048] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.002s
I20260321 02:59:09.927431 13099 consensus_queue.cc:1048] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260321 02:59:09.945441 12924 raft_consensus.cc:2804] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:09.929440 12856 consensus_queue.cc:1048] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260321 02:59:09.978134 11200 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:09.979424 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.979439 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:09.983376 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.984265 11306 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:09.985915 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:09.961999 13123 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:09.986760 11310 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:09.981338 13094 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:10.006356 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:10.008049 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:10.010073 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:10.010811 11310 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:09.988397 12961 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:10.020687 11306 raft_consensus.cc:2468] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:59:09.990053 13009 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:09.991822 12924 raft_consensus.cc:697] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:09.989601 13128 raft_consensus.cc:2749] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:10.007238 13094 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:10.007205 13123 raft_consensus.cc:697] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:10.070386 11306 raft_consensus.cc:1240] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3. Ops: []
I20260321 02:59:10.020000 13097 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:10.020928 12720 raft_consensus.cc:3055] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:59:10.014406 13084 consensus_queue.cc:1048] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20260321 02:59:10.069139 13094 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:10.077003 12720 raft_consensus.cc:740] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
W20260321 02:59:10.122944 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 136944us
W20260321 02:59:10.118904 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 129136us
I20260321 02:59:10.072269 13009 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:10.080418 13097 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:10.072716 12736 consensus_queue.cc:237] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:10.120374 12968 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:10.117857 13131 raft_consensus.cc:2749] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:10.138490 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:10.145102 11303 raft_consensus.cc:1275] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 02:59:10.146612 11303 raft_consensus.cc:1275] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:10.161062 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:10.130853 13129 consensus_queue.cc:1048] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260321 02:59:10.175288 11308 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:10.139158 12924 consensus_queue.cc:237] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:10.140919 13076 consensus_queue.cc:1048] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.042s
I20260321 02:59:10.148808 12968 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:10.185859 12968 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:10.189762 13009 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:10.191133 13128 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:10.191708 13128 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:10.194164 13128 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:10.198369 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:10.199366 11977 raft_consensus.cc:2410] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260321 02:59:10.168994 13130 consensus_queue.cc:1059] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260321 02:59:10.204156 13094 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:10.205085 13130 raft_consensus.cc:3055] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:10.205502 13130 raft_consensus.cc:740] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:10.206848 13094 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:10.206404 13130 consensus_queue.cc:260] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:10.133319 12856 consensus_queue.cc:1048] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20260321 02:59:10.209160 13097 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:10.141256 12934 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:10.213409 11376 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:10.171062 13123 consensus_queue.cc:237] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:10.216334 12848 raft_consensus.cc:3055] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Stepping down as leader of term 2
W20260321 02:59:10.217880 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 74413.7us
I20260321 02:59:10.218252 11304 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:59:10.195933 11310 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
W20260321 02:59:10.220871 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 76240.1us
I20260321 02:59:10.221541 11200 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:10.140347 12961 raft_consensus.cc:697] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:10.151966 13093 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:10.217128 12848 raft_consensus.cc:740] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:10.231063 13103 consensus_queue.cc:1048] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:59:10.231881 13096 consensus_queue.cc:1048] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:10.232052 12997 consensus_queue.cc:1048] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:10.232831 13103 consensus_queue.cc:1048] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:59:10.235762 11200 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:10.236549 13133 raft_consensus.cc:2749] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:10.239158 13096 consensus_queue.cc:1048] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:10.239519 13103 consensus_queue.cc:1048] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:10.240634 11200 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:10.242076 11200 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:10.242810 13096 raft_consensus.cc:2804] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:10.243767 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:10.243638 13096 raft_consensus.cc:697] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:10.245256 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:10.245507 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:10.246096 11227 raft_consensus.cc:2393] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 3: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:10.246640 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:10.247283 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:10.247702 11233 raft_consensus.cc:1240] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 2. Current term is 3. Ops: []
I20260321 02:59:10.248001 11235 raft_consensus.cc:2468] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:59:10.248103 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:10.248883 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:10.249061 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:10.249949 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:10.250362 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:10.250553 11235 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:59:10.251276 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:10.251314 11228 raft_consensus.cc:2393] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:10.252149 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:10.252173 11310 raft_consensus.cc:2410] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260321 02:59:10.247578 11236 raft_consensus.cc:2393] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:10.254266 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:10.250375 11232 raft_consensus.cc:1275] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:10.255681 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:10.256733 11227 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:10.262840 13128 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:10.263401 13128 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:10.265318 11200 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:10.266623 13099 raft_consensus.cc:2749] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:10.265537 13128 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:10.267783 13096 consensus_queue.cc:237] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:10.281944 11349 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:10.222455 12987 raft_consensus.cc:2749] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:10.225598 12846 raft_consensus.cc:2749] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:10.284574 11349 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:10.287159 13137 raft_consensus.cc:2804] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:10.161578 12720 consensus_queue.cc:260] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:10.281651 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:10.326357 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 357555us
I20260321 02:59:10.277436 13129 consensus_queue.cc:1048] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.089s
I20260321 02:59:10.344151 11275 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:10.293220 13137 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:10.278273 13094 consensus_queue.cc:1059] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20260321 02:59:10.301262 13092 consensus_queue.cc:1048] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20260321 02:59:10.301062 12968 raft_consensus.cc:2804] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:10.356259 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:10.289296 13093 consensus_queue.cc:1059] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20260321 02:59:10.356787 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:10.369239 11376 raft_consensus.cc:2368] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4.
I20260321 02:59:10.322595 11227 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:10.371088 11275 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:10.371639 11349 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:10.356967 12934 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:10.337769 12880 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:10.424588 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 53568.3us
I20260321 02:59:10.390056 13084 raft_consensus.cc:2749] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:10.431869 13135 raft_consensus.cc:2749] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:10.434293 13144 raft_consensus.cc:2749] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:10.363590 13137 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:10.436192 13139 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:10.437120 13139 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:10.438215 11200 leader_election.cc:400] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Vote denied by peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533) with higher term. Message: Invalid argument: T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4.
I20260321 02:59:10.438580 11200 leader_election.cc:403] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260321 02:59:10.362485 12968 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:10.453531 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:10.456030 11307 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:10.440186 12968 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:10.469388 11307 raft_consensus.cc:1275] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260321 02:59:10.471448 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:10.479663 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 108102us
W20260321 02:59:10.491429 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 45254.6us
I20260321 02:59:10.439347 13139 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:10.493973 11200 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:10.445657 13093 consensus_queue.cc:1243] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer 0d7efc590a1041a9bd4cd4a971332dfb log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260321 02:59:10.503052 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:10.516185 11307 raft_consensus.cc:2393] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 3: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:10.503625 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:10.517089 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:10.476643 12976 raft_consensus.cc:3055] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:59:10.366366 12848 consensus_queue.cc:260] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:10.516888 11382 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:10.515360 11349 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:10.450235 13099 raft_consensus.cc:2749] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533) with higher term. Message: Invalid argument: T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4.
I20260321 02:59:10.530710 11230 raft_consensus.cc:1275] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:10.375916 12961 consensus_queue.cc:237] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:10.542723 11307 raft_consensus.cc:1275] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:59:10.453078 13123 consensus_queue.cc:1048] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.068s
I20260321 02:59:10.551473 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:10.555210 11382 raft_consensus.cc:2393] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 4: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:10.522445 13129 raft_consensus.cc:2804] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:10.460959 12880 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:10.559303 11349 leader_election.cc:400] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Vote denied by peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267) with higher term. Message: Invalid argument: T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5.
I20260321 02:59:10.523663 12976 raft_consensus.cc:740] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:10.530109 12856 consensus_queue.cc:1048] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.048s
I20260321 02:59:10.559837 13129 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:10.584746 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:10.586571 13086 consensus_queue.cc:1048] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.002s
W20260321 02:59:10.587380 13148 log.cc:927] Time spent T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.063s user 0.001s sys 0.000s
I20260321 02:59:10.586592 13159 raft_consensus.cc:2749] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:10.596841 11307 raft_consensus.cc:1275] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:10.601787 11349 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:10.603096 11349 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:10.584338 13143 raft_consensus.cc:2804] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:10.583372 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:10.622340 13152 log.cc:927] Time spent T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20260321 02:59:10.592856 13129 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:10.631477 11349 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:10.632326 11378 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:10.605903 13093 raft_consensus.cc:2749] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:10.641916 11349 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:10.631991 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 149069us
W20260321 02:59:10.666759 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 132200us
I20260321 02:59:10.691418 11230 raft_consensus.cc:1275] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:10.693522 13123 consensus_queue.cc:1048] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:59:10.693593 13086 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:10.630878 12976 consensus_queue.cc:260] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:10.698982 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:10.642335 13143 raft_consensus.cc:697] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:10.695364 13138 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:10.730947 11349 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
W20260321 02:59:10.713187 13152 log.cc:927] Time spent T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.070s user 0.002s sys 0.002s
W20260321 02:59:10.740778 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:10.699653 13162 raft_consensus.cc:3055] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:59:10.694572 13086 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:10.710425 13093 raft_consensus.cc:2749] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:10.758811 13165 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:10.759186 12880 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:10.759317 13165 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:10.760063 13165 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:10.760885 12880 consensus_queue.cc:1059] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:59:10.762063 12880 raft_consensus.cc:2804] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:59:10.762576 12880 raft_consensus.cc:697] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:10.715276 11972 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:10.752395 13092 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:10.771476 13156 log.cc:927] Time spent T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.144s user 0.001s sys 0.000s
W20260321 02:59:10.771221 13147 log.cc:927] Time spent T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.141s user 0.000s sys 0.001s
I20260321 02:59:10.771180 13158 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:10.746163 13162 raft_consensus.cc:740] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:10.797416 11234 raft_consensus.cc:1240] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6. Ops: []
W20260321 02:59:10.809005 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 62354.3us
I20260321 02:59:10.773591 13094 consensus_queue.cc:1048] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260321 02:59:10.887840 11276 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:10.763536 12880 consensus_queue.cc:237] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:10.906620 13151 log.cc:927] Time spent T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.218s user 0.001s sys 0.000s
I20260321 02:59:10.906800 13162 consensus_queue.cc:260] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:10.907541 13143 consensus_queue.cc:237] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:10.887478 13152 log.cc:927] Time spent T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
W20260321 02:59:10.913633 13150 log.cc:927] Time spent T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.172s user 0.000s sys 0.000s
W20260321 02:59:10.915660 13160 log.cc:927] Time spent T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.199s user 0.000s sys 0.001s
I20260321 02:59:10.916090 13135 raft_consensus.cc:2804] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:10.916755 13135 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:10.917259 13135 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:10.917906 13086 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:10.920544 13172 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:10.921090 13172 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:10.908471 13162 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
W20260321 02:59:10.929646 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 198809us
I20260321 02:59:10.938172 11349 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:10.942589 11349 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:10.771790 13092 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:10.954818 11275 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:10.971837 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 222545us
I20260321 02:59:11.070922 11349 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:11.075299 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 120569us
I20260321 02:59:10.947289 12987 raft_consensus.cc:2749] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:11.138124 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:10.889171 13158 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.121923 13139 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:11.186103 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 298022us
W20260321 02:59:11.192313 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 121595us
W20260321 02:59:11.134438 13096 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.040s sys 0.130s
W20260321 02:59:11.219411 13096 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.658s user 0.040s sys 0.131s
W20260321 02:59:11.218446 13155 log.cc:927] Time spent T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.519s user 0.001s sys 0.000s
I20260321 02:59:11.213680 13086 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:11.220805 13145 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:11.251046 13123 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:11.218946 13139 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.248144 13185 consensus_queue.cc:1048] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.081s
I20260321 02:59:11.291549 13171 raft_consensus.cc:2749] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:11.136847 12934 raft_consensus.cc:2749] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:11.288193 13109 consensus_queue.cc:1059] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.049s
W20260321 02:59:11.320640 12968 consensus_meta.cc:287] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.822s user 0.008s sys 0.005s
I20260321 02:59:11.292766 12880 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:11.288879 13086 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:11.408118 13109 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:11.408797 13182 log.cc:927] Time spent T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20260321 02:59:11.408712 13109 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.410341 13133 raft_consensus.cc:3055] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:11.411048 13133 raft_consensus.cc:740] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:11.411405 13109 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:11.412608 13174 raft_consensus.cc:2955] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } } }
I20260321 02:59:11.320858 13123 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:11.292219 13145 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.422761 12968 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.424006 12880 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:11.402258 13183 log.cc:927] Time spent T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.270s user 0.000s sys 0.001s
I20260321 02:59:11.430208 13171 raft_consensus.cc:2749] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260321 02:59:11.452348 13179 log.cc:927] Time spent T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.474s user 0.002s sys 0.000s
I20260321 02:59:11.452764 13171 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:11.431093 13198 raft_consensus.cc:3055] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:11.432006 13199 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:11.432663 13200 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:11.433135 13172 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:11.433403 13092 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:11.434732 13086 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:11.435247 13163 log.cc:927] Time spent T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.172s user 0.001s sys 0.000s
I20260321 02:59:11.435603 13185 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:11.436053 11377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1401) took 1106 ms. Trace:
I20260321 02:59:11.436511 13158 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:11.438401 11306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1407) took 1177 ms. Trace:
I20260321 02:59:11.438815 13195 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:11.439390 11311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56616 (request call id 1515) took 1488 ms (client timeout 1637 ms). Trace:
I20260321 02:59:11.428527 13133 consensus_queue.cc:260] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:11.476477 11942 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.818s user 0.010s sys 0.004s
I20260321 02:59:11.442215 13145 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:11.442763 13186 log.cc:927] Time spent T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260321 02:59:11.444442 13123 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:11.445281 13190 log.cc:927] Time spent T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
W20260321 02:59:11.453691 11350 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.721s user 0.002s sys 0.015s
W20260321 02:59:11.525008 11350 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.792s user 0.002s sys 0.015s
W20260321 02:59:11.525426 11350 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 806115us
W20260321 02:59:11.387652 13137 consensus_meta.cc:287] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.943s user 0.014s sys 0.003s
W20260321 02:59:11.448460 13165 consensus_meta.cc:287] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.679s user 0.009s sys 0.000s
I20260321 02:59:11.452337 13192 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:11.476418 11311 rpcz_store.cc:269] 0321 02:59:09.951163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:09.951723 (+ 560us) service_pool.cc:225] Handling call
0321 02:59:11.416473 (+1464750us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b54000e88d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:11.439334 (+ 22861us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":615339904}
W20260321 02:59:11.478255 11942 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.819s user 0.011s sys 0.004s
I20260321 02:59:11.448892 13109 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:11.458207 12968 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:11.458113 13198 raft_consensus.cc:740] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:11.480106 13133 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:11.480556 11382 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1407) took 1084 ms. Trace:
I20260321 02:59:11.480484 13195 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:11.566403 13195 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:11.569785 13135 consensus_meta.cc:287] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.652s user 0.012s sys 0.000s
I20260321 02:59:11.483776 12880 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:11.453401 13171 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.484135 13158 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:11.484442 13194 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:11.484869 13162 consensus_meta.cc:287] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.539s user 0.009s sys 0.000s
I20260321 02:59:11.484508 13185 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.476250 11306 rpcz_store.cc:276] 0321 02:59:10.261481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:10.261607 (+ 126us) service_pool.cc:225] Handling call
0321 02:59:10.261974 (+ 367us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:10.288956 (+ 26982us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:10.289007 (+ 51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:10.300773 (+ 11766us) log.cc:844] Serialized 26 byte log entry
0321 02:59:10.765588 (+464815us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:59:10.765634 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:10.765711 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:11.438158 (+672447us) raft_consensus.cc:1685] finished
0321 02:59:11.438184 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:11.438333 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28667,"raft.queue_time_us":58818,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":21793,"thread_start_us":402717,"threads_started":1,"wal-append.queue_time_us":1044158}
I20260321 02:59:11.478775 13200 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:11.470775 13199 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:11.489702 13140 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:11.490018 11379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1403) took 1150 ms. Trace:
I20260321 02:59:11.617841 11379 rpcz_store.cc:276] 0321 02:59:10.340072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:10.340195 (+ 123us) service_pool.cc:225] Handling call
0321 02:59:10.340572 (+ 377us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:10.417531 (+ 76959us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:10.417583 (+ 52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:10.418383 (+ 800us) log.cc:844] Serialized 26 byte log entry
0321 02:59:10.531937 (+113554us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:10.532097 (+ 160us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:10.532198 (+ 101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:11.489791 (+957593us) raft_consensus.cc:1685] finished
0321 02:59:11.489816 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:11.489958 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27205,"raft.queue_time_us":150190,"spinlock_wait_cycles":30708992,"thread_start_us":144622,"threads_started":2,"wal-append.queue_time_us":636807}
W20260321 02:59:11.491971 13129 consensus_meta.cc:287] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.820s user 0.009s sys 0.000s
W20260321 02:59:11.494778 11378 consensus_meta.cc:287] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.814s user 0.010s sys 0.000s
W20260321 02:59:11.496271 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:11.550595 13193 log.cc:927] Time spent T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.156s user 0.000s sys 0.001s
I20260321 02:59:11.541229 13165 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.551355 13169 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:11.537009 13137 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.563625 13123 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:11.520867 13144 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:11.570360 13135 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:11.555356 11382 rpcz_store.cc:276] 0321 02:59:10.396143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:10.586737 (+190594us) service_pool.cc:225] Handling call
0321 02:59:10.587080 (+ 343us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:10.598118 (+ 11038us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:10.598153 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:10.598815 (+ 662us) log.cc:844] Serialized 26 byte log entry
0321 02:59:10.688284 (+ 89469us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:10.688324 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:10.688397 (+ 73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:11.480326 (+791929us) raft_consensus.cc:1685] finished
0321 02:59:11.480347 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:11.480496 (+ 149us) inbound_call.cc:173] Queueing success response
0321 02:59:11.492618 (+ 12122us) spinlock_profiling.cc:243] Waited 2.31 ms on lock 0x7b48001ec258. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b14721a1 00007fa8b1469237 00007fa8b14690ca 00007fa8b156092f 00007fa8b15605f7 00007fa8b1574c1c 00007fa8b1574bc8 00007fa8b1574b48 00007fa8b1574b03 00007fa8b1573dac 00007fa8b04acdf6
Metrics: {"mutex_wait_us":6215,"raft.queue_time_us":8809,"raft.run_cpu_time_us":5348,"raft.run_wall_time_us":886216,"spinlock_wait_cycles":5321600,"thread_start_us":54752,"threads_started":1,"wal-append.queue_time_us":264213}
W20260321 02:59:11.586927 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:11.587218 13145 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:11.604014 11383 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.710s user 0.017s sys 0.024s
I20260321 02:59:11.607792 11349 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:11.475809 11377 rpcz_store.cc:276] 0321 02:59:10.329984 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:10.330672 (+ 688us) service_pool.cc:225] Handling call
0321 02:59:10.331103 (+ 431us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:10.331582 (+ 479us) raft_consensus.cc:1534] Early marking committed up to index 2
0321 02:59:10.331623 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:10.332245 (+ 622us) log.cc:844] Serialized 26 byte log entry
0321 02:59:10.469028 (+136783us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:10.469103 (+ 75us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:10.469209 (+ 106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:11.435793 (+966584us) raft_consensus.cc:1685] finished
0321 02:59:11.435816 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:11.435939 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":116442,"raft.queue_time_us":11601,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":20861,"thread_start_us":112405,"threads_started":1,"wal-append.queue_time_us":229325}
I20260321 02:59:11.622220 11304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1404) took 1361 ms. Trace:
I20260321 02:59:11.625540 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:11.628500 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:11.629874 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:11.632931 11229 raft_consensus.cc:1240] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260321 02:59:11.634263 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:11.636998 13171 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:11.556581 13192 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.639319 13200 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:11.644843 11378 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:11.647012 13199 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:11.649010 13185 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:11.596484 13194 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:11.588186 13158 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:11.677609 13162 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:11.551771 13109 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.621176 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:11.714537 11236 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:11.673028 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:11.678069 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:11.679009 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:11.669173 13169 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.679414 13144 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.680274 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:11.681131 13137 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:11.676391 13123 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:11.682228 13165 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:11.662540 13129 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.658026 13140 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:11.621030 13208 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:11.684404 11230 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
W20260321 02:59:11.688675 11383 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.795s user 0.017s sys 0.025s
I20260321 02:59:11.689397 11304 rpcz_store.cc:276] 0321 02:59:10.260451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:10.262171 (+ 1720us) service_pool.cc:225] Handling call
0321 02:59:10.262555 (+ 384us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:10.319739 (+ 57184us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:10.319802 (+ 63us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:10.320458 (+ 656us) log.cc:844] Serialized 26 byte log entry
0321 02:59:10.757941 (+437483us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:10.758329 (+ 388us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:10.765469 (+ 7140us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:11.622017 (+856548us) raft_consensus.cc:1685] finished
0321 02:59:11.622036 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:11.622164 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28374,"raft.queue_time_us":79596,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":17276,"thread_start_us":377790,"threads_started":1,"wal-append.queue_time_us":832115}
I20260321 02:59:11.694958 11231 raft_consensus.cc:2393] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 3: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:11.704072 11377 raft_consensus.cc:1275] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260321 02:59:11.761317 11377 pending_rounds.cc:85] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5: Aborting all ops after (but not including) 1
I20260321 02:59:11.713587 11382 raft_consensus.cc:1240] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 2. Current term is 3. Ops: []
W20260321 02:59:11.656364 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 356211us
W20260321 02:59:11.717304 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 109622us
I20260321 02:59:11.715911 11233 raft_consensus.cc:2468] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:11.720958 11228 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:11.623383 13198 consensus_queue.cc:260] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.724150 13207 raft_consensus.cc:2749] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:11.724427 13158 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:11.724640 13192 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:11.726341 13194 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:11.691432 13139 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:11.781236 11383 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1404) took 1440 ms. Trace:
I20260321 02:59:11.781507 12987 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:11.782060 12987 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.782519 13129 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:11.783739 13123 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:11.726804 11376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1413) took 1326 ms. Trace:
I20260321 02:59:11.785224 12987 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:11.784343 11376 rpcz_store.cc:276] 0321 02:59:10.400522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:11.074695 (+674173us) service_pool.cc:225] Handling call
0321 02:59:11.137107 (+ 62412us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:11.272578 (+135471us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:11.272636 (+ 58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:11.291930 (+ 19294us) log.cc:844] Serialized 26 byte log entry
0321 02:59:11.555404 (+263474us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:11.555445 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:11.555583 (+ 138us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:11.726553 (+170970us) raft_consensus.cc:1685] finished
0321 02:59:11.726576 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:11.726725 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":274908,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":152,"thread_start_us":242567,"threads_started":2,"wal-append.queue_time_us":376090}
I20260321 02:59:11.786818 11376 raft_consensus.cc:1275] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:11.787190 13138 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:11.787379 11376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40950 (request call id 1064) took 1304 ms. Trace:
I20260321 02:59:11.708936 13011 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
W20260321 02:59:11.775028 12990 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.192s user 0.015s sys 0.032s
I20260321 02:59:11.791685 13011 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:11.792480 12990 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.212s user 0.015s sys 0.033s
I20260321 02:59:11.794378 13011 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:11.781836 13109 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:11.708395 13202 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:11.760335 11230 raft_consensus.cc:606] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:11.761605 11377 pending_rounds.cc:107] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260321 02:59:11.800163 11377 raft_consensus.cc:2889] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:11.801116 11377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1416) took 1328 ms. Trace:
I20260321 02:59:11.801546 11377 rpcz_store.cc:276] 0321 02:59:10.476235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:11.691872 (+1215637us) service_pool.cc:225] Handling call
0321 02:59:11.693801 (+ 1929us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:11.800770 (+106969us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:11.801041 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:11.762097 11382 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1415) took 1360 ms. Trace:
W20260321 02:59:11.753525 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 200660us
I20260321 02:59:11.730527 13200 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:11.728480 11378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40934 (request call id 1412) took 1317 ms. Trace:
I20260321 02:59:11.755200 13097 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:11.730542 11972 raft_consensus.cc:2468] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:11.766008 13208 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.695441 13145 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:11.787905 11376 rpcz_store.cc:276] 0321 02:59:10.482512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:11.786099 (+1303587us) service_pool.cc:225] Handling call
0321 02:59:11.786450 (+ 351us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:11.787151 (+ 701us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:11.787328 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:11.722173 13162 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.749215 13185 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:11.772969 13165 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:11.802664 11382 rpcz_store.cc:276] 0321 02:59:10.401884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:11.689840 (+1287956us) service_pool.cc:225] Handling call
0321 02:59:11.713399 (+ 23559us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:11.761819 (+ 48420us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:11.762040 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260321 02:59:11.824076 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 55466.5us
W20260321 02:59:11.803279 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 104270us
I20260321 02:59:11.799731 11230 raft_consensus.cc:614] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:11.781729 11383 rpcz_store.cc:276] 0321 02:59:10.341355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:10.341506 (+ 151us) service_pool.cc:225] Handling call
0321 02:59:10.341922 (+ 416us) raft_consensus.cc:1471] Updating replica for 2 ops
0321 02:59:10.443881 (+101959us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:10.443929 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0321 02:59:10.470145 (+ 26216us) log.cc:844] Serialized 52 byte log entry
0321 02:59:10.589745 (+119600us) spinlock_profiling.cc:243] Waited 19 ms on lock 0x7b48001ec258. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b14721a1 00007fa8b1469237 00007fa8b14690ca 00007fa8b156092f 00007fa8b15605f7 00007fa8b1574c1c 00007fa8b1574bc8 00007fa8b1574b48 00007fa8b1574b03 00007fa8b1573dac 00007fa8b04acdf6
0321 02:59:10.765515 (+175770us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:11.760432 (+994917us) log.cc:844] Serialized 14 byte log entry
0321 02:59:11.774566 (+ 14134us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:11.774658 (+ 92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:11.774700 (+ 42us) raft_consensus.cc:1685] finished
0321 02:59:11.780963 (+ 6263us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:11.781174 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13831,"prepare.queue_time_us":643198,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":32,"raft.queue_time_us":73031,"raft.run_cpu_time_us":1829,"raft.run_wall_time_us":148434,"spinlock_wait_cycles":51124480,"thread_start_us":1160477,"threads_started":3,"wal-append.queue_time_us":1160974}
I20260321 02:59:11.787776 13138 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:11.808063 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:11.778083 13158 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:11.853086 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:11.815048 13123 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:11.857306 11383 raft_consensus.cc:2468] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:11.860402 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:11.861210 11379 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:11.814791 13139 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:11.863902 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:11.859798 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:11.865057 11382 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:11.865410 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:11.867889 11383 raft_consensus.cc:2468] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 2.
I20260321 02:59:11.869885 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:11.823390 13109 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:11.815088 13097 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.805416 11378 rpcz_store.cc:276] 0321 02:59:10.399794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:10.560804 (+161010us) service_pool.cc:225] Handling call
0321 02:59:11.716914 (+1156110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:11.832731 13165 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.861215 13086 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:11.815130 13200 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.864804 12987 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:11.884928 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:11.863327 13158 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.833339 13185 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.873518 13210 consensus_queue.cc:1059] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.025s
I20260321 02:59:11.875487 13109 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.888248 13208 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:11.898278 13109 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:11.885457 12934 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:11.808867 13202 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.899210 12934 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.890477 13196 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:11.900368 13196 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:11.901230 13202 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:11.893184 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:11.893533 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:11.894248 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:11.892426 11230 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:11.903340 11230 raft_consensus.cc:606] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:11.904492 11376 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:11.905390 11942 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:11.891824 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:11.912670 13213 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:11.913383 13165 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:11.913503 13204 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:11.918205 13097 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:11.919320 13144 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:11.920469 13145 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:11.922533 13200 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:11.901448 12934 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:11.869372 13123 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.923059 11377 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
W20260321 02:59:11.940311 13190 log.cc:927] Time spent T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260321 02:59:11.894081 13216 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:11.872704 13139 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:11.890700 12987 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:11.881371 13135 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:11.939211 13217 log.cc:927] Time spent T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
W20260321 02:59:11.949257 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 43986.6us
I20260321 02:59:11.953655 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:11.902417 13196 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:11.954257 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:11.955322 13011 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:11.987303 13011 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:11.988155 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:11.889730 13086 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:11.969508 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:11.953015 13194 raft_consensus.cc:2804] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:12.001830 13123 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:12.002362 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:12.003444 13220 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:12.004087 13220 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:12.005134 13086 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:11.997864 13194 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:12.006033 13194 raft_consensus.cc:3060] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:12.006250 13220 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:11.981050 13165 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:11.965440 13169 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:11.971570 13200 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:11.987933 13226 consensus_queue.cc:1048] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260321 02:59:11.955413 13162 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:11.989468 11382 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:11.976099 13216 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:12.013916 13139 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:11.961093 13213 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:12.014722 13097 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:12.015377 12987 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:11.976187 13140 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:12.051002 13097 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:12.053424 13097 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:12.053964 13138 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:12.055449 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:12.057262 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:11.977994 11383 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:11.956377 13204 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:12.003705 11971 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45098 (request call id 299) took 1463 ms. Trace:
I20260321 02:59:12.038954 13011 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:12.040766 13135 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:12.039433 11979 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:12.010186 13086 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:12.038517 12848 raft_consensus.cc:3060] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:12.047971 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:12.050248 13213 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:12.059264 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:12.062045 13084 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:12.034564 13158 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:12.066001 11979 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:59:12.063526 11971 rpcz_store.cc:276] 0321 02:59:10.540541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:10.582206 (+ 41665us) service_pool.cc:225] Handling call
0321 02:59:12.002710 (+1420504us) spinlock_profiling.cc:243] Waited 5.44 ms on lock 0x7b58002c3798. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b1522680 00007fa8b150cfe9 00007fa8b150ca92 00007fa8aaa8ad34
0321 02:59:12.002982 (+ 272us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:12.003186 (+ 204us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:12.003219 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:12.003277 (+ 58us) raft_consensus.cc:1645] Marking committed up to 5
0321 02:59:12.003318 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:12.003387 (+ 69us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:12.003513 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12504448}
I20260321 02:59:12.072901 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:12.085996 11383 raft_consensus.cc:2468] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 5.
I20260321 02:59:12.027189 13165 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:12.089661 11942 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:12.067691 13216 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:12.107153 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:12.045202 13200 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:12.113600 11383 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1418) took 1392 ms. Trace:
I20260321 02:59:12.112915 11350 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:12.118364 13213 raft_consensus.cc:2804] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:12.118950 13213 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:12.119131 13204 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:12.119390 13213 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:12.090927 13138 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:12.121340 13165 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:12.122059 13158 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:12.122632 13158 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:12.123571 11379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1417) took 1627 ms. Trace:
I20260321 02:59:12.123943 11379 rpcz_store.cc:276] 0321 02:59:10.496792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:12.111564 (+1614772us) service_pool.cc:225] Handling call
0321 02:59:12.123065 (+ 11501us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:12.123249 (+ 184us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:12.123273 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:12.123316 (+ 43us) raft_consensus.cc:1645] Marking committed up to 3
0321 02:59:12.123345 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:12.123398 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:12.123512 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:12.124799 13158 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:12.113503 13154 raft_consensus.cc:2804] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:12.126037 11383 rpcz_store.cc:276] 0321 02:59:10.721513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:12.112453 (+1390940us) service_pool.cc:225] Handling call
0321 02:59:12.113100 (+ 647us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:12.113268 (+ 168us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:12.113294 (+ 26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:12.113339 (+ 45us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:12.113366 (+ 27us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:12.113420 (+ 54us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:12.113541 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:12.125720 11379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1419) took 1403 ms. Trace:
I20260321 02:59:12.129662 11383 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1420) took 1407 ms. Trace:
I20260321 02:59:12.073128 13135 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:12.072686 13086 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:12.130425 11383 rpcz_store.cc:276] 0321 02:59:10.722245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:12.128187 (+1405942us) service_pool.cc:225] Handling call
0321 02:59:12.128728 (+ 541us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:12.129067 (+ 339us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:12.129219 (+ 152us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:12.129304 (+ 85us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:12.129364 (+ 60us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:12.129448 (+ 84us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:12.129590 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:12.140290 11305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56616 (request call id 1524) took 1021 ms. Trace:
I20260321 02:59:12.138367 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:12.148883 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:12.147511 11305 rpcz_store.cc:276] 0321 02:59:11.118751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:11.131363 (+ 12612us) service_pool.cc:225] Handling call
0321 02:59:11.131834 (+ 471us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:11.399518 (+267684us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:11.399548 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:11.399591 (+ 43us) raft_consensus.cc:1645] Marking committed up to 5
0321 02:59:11.399622 (+ 31us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:12.139993 (+740371us) spinlock_profiling.cc:243] Waited 267 ms on lock 0x7b54002d1754. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:12.140041 (+ 48us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:12.140223 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":615088512}
I20260321 02:59:12.086153 13084 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:12.116834 13185 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:12.129856 11379 rpcz_store.cc:276] 0321 02:59:10.721837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:12.124892 (+1403055us) service_pool.cc:225] Handling call
0321 02:59:12.125199 (+ 307us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:12.125393 (+ 194us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:12.125416 (+ 23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:12.125455 (+ 39us) raft_consensus.cc:1645] Marking committed up to 3
0321 02:59:12.125493 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:12.125546 (+ 53us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:12.125643 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:12.152773 13191 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:12.158133 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:12.125720 13154 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:12.133991 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:12.161602 13194 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:12.163785 13189 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:12.135347 13200 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:12.166653 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 291407us
I20260321 02:59:12.136647 13234 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:12.164283 13084 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:12.172034 11379 raft_consensus.cc:1275] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:12.176508 11383 raft_consensus.cc:1240] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:12.167934 11275 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:12.165263 11971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:12.182530 11275 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:12.166330 13154 raft_consensus.cc:3060] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:12.165761 13191 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:12.322917 13191 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:12.326846 13135 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:12.328907 13194 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:12.331600 13086 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:12.318137 13084 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:12.332880 13084 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:12.318140 13192 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:12.182893 12924 raft_consensus.cc:2804] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:12.336642 12924 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:12.337031 12924 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
W20260321 02:59:12.340818 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 158368us
I20260321 02:59:12.335566 13084 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:12.376468 11232 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
W20260321 02:59:12.378230 11232 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58214 (request call id 1082) took 1880 ms (client timeout 1658 ms). Trace:
I20260321 02:59:12.379035 11275 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:12.378899 11232 rpcz_store.cc:269] 0321 02:59:10.498106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:10.498344 (+ 238us) service_pool.cc:225] Handling call
0321 02:59:12.138189 (+1639845us) spinlock_profiling.cc:243] Waited 1.17 ms on lock 0x7b5800111398. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b150e760 00007fa8b150e044 00007fa8aaa8f834 00007fa8aaa8f7c2
0321 02:59:12.378130 (+239941us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2681600}
I20260321 02:59:12.381886 11383 raft_consensus.cc:1275] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:12.385557 12846 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:12.396274 13229 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:12.396430 12846 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:12.396871 13229 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:12.397640 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:12.320056 11380 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40950 (request call id 1063) took 1837 ms (client timeout 1716 ms). Trace:
I20260321 02:59:12.400108 12856 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
W20260321 02:59:12.399919 11380 rpcz_store.cc:269] 0321 02:59:10.482242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:10.570443 (+ 88201us) service_pool.cc:225] Handling call
0321 02:59:12.133738 (+1563295us) spinlock_profiling.cc:243] Waited 2.08 ms on lock 0x7b58002e3298. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a947917c 00007fa8a9478d3b 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b150e760 00007fa8b150e044 00007fa8aaa8f834 00007fa8aaa8f7c2
0321 02:59:12.207974 (+ 74236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4777216}
I20260321 02:59:12.400997 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:12.400816 12856 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:12.402143 13169 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:12.402061 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:12.402726 13169 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:12.388602 13086 raft_consensus.cc:2804] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:12.404331 12846 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:12.404413 13086 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:12.406859 11275 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:12.415115 11384 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1408) took 2018 ms. Trace:
I20260321 02:59:12.421168 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:12.422945 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:12.423708 11311 raft_consensus.cc:2468] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:12.426342 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:12.422135 11305 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:12.426960 11311 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:12.336654 13192 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:12.428956 11979 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:12.461865 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:12.405150 13086 raft_consensus.cc:3060] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:12.499414 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:12.501567 11311 raft_consensus.cc:2410] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260321 02:59:12.169659 13189 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:12.510600 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:12.512748 11276 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.638s user 0.125s sys 0.225s
W20260321 02:59:12.513197 11276 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.638s user 0.125s sys 0.225s
W20260321 02:59:12.513518 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 639041us
I20260321 02:59:12.511533 11304 raft_consensus.cc:2468] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:12.340519 13195 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:12.534600 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:12.538656 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:12.404624 13169 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:12.178521 13234 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:12.540977 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:12.541795 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:12.418656 12990 raft_consensus.cc:2804] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:12.543131 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:12.544090 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:12.621204 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:12.546298 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:12.572908 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:12.404969 12846 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:12.635322 11233 raft_consensus.cc:1240] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: []
W20260321 02:59:12.587613 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 46028.8us
I20260321 02:59:12.594341 11311 raft_consensus.cc:2393] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 3: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:12.637630 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:12.595810 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:12.415858 11384 rpcz_store.cc:276] 0321 02:59:10.397090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:10.880971 (+483881us) service_pool.cc:225] Handling call
0321 02:59:10.901978 (+ 21007us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:11.466721 (+564743us) raft_consensus.cc:1534] Early marking committed up to index 2
0321 02:59:11.520084 (+ 53363us) log.cc:844] Serialized 14 byte log entry
0321 02:59:11.788349 (+268265us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:11.791817 (+ 3468us) log.cc:844] Serialized 26 byte log entry
0321 02:59:11.792018 (+ 201us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:11.792050 (+ 32us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:12.374862 (+582812us) spinlock_profiling.cc:243] Waited 292 ms on lock 0x7b5400099554. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:12.375029 (+ 167us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:12.375129 (+ 100us) raft_consensus.cc:1685] finished
0321 02:59:12.414553 (+ 39424us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:12.415015 (+ 462us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":99089,"prepare.queue_time_us":53148,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":4798,"raft.queue_time_us":263107,"raft.run_cpu_time_us":517,"raft.run_wall_time_us":13005,"spinlock_wait_cycles":671528192,"thread_start_us":385106,"threads_started":2,"wal-append.queue_time_us":475420}
I20260321 02:59:12.540977 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:12.418956 13135 raft_consensus.cc:2804] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:12.617014 11308 raft_consensus.cc:1275] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:59:12.541255 13185 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:12.429051 13235 consensus_queue.cc:1048] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.237s
I20260321 02:59:12.645937 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:12.717568 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:12.786952 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:12.758031 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:12.902457 11942 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.719s user 0.006s sys 0.019s
W20260321 02:59:12.909291 11942 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.726s user 0.006s sys 0.019s
I20260321 02:59:12.913442 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:12.917255 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:12.919592 11309 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:59:12.533233 13084 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:12.811781 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:12.905800 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:12.929054 11311 raft_consensus.cc:2393] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 4: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:12.929073 11233 raft_consensus.cc:2393] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 4: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
W20260321 02:59:12.907073 11381 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.862s user 0.001s sys 0.040s
W20260321 02:59:12.930224 11381 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.885s user 0.001s sys 0.040s
I20260321 02:59:12.931877 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:12.932749 11309 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:12.608012 13123 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:12.829689 13203 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.773s user 0.009s sys 0.057s
W20260321 02:59:12.853440 13128 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.265s user 0.002s sys 0.026s
W20260321 02:59:12.646013 13096 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.723s user 0.032s sys 0.028s
W20260321 02:59:12.888804 13143 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.708s user 0.001s sys 0.021s
I20260321 02:59:12.619128 12990 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:12.859450 12961 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.706s user 0.010s sys 0.021s
I20260321 02:59:12.934850 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:12.956622 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:12.962091 11231 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:12.962226 11311 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:12.692636 13100 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:12.690207 13138 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:12.690052 13216 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:12.888182 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:12.720981 13174 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.741s user 0.032s sys 0.032s
I20260321 02:59:12.862211 12856 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:13.006441 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:13.025133 11229 raft_consensus.cc:2393] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
W20260321 02:59:12.924477 13129 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.049s user 0.009s sys 0.020s
I20260321 02:59:12.690412 13165 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:12.689932 13097 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:12.664467 12846 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:12.945405 13128 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.357s user 0.003s sys 0.028s
I20260321 02:59:12.955601 11235 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:59:12.923136 13185 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:12.845888 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:12.935665 13213 consensus_meta.cc:287] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.816s user 0.008s sys 0.009s
W20260321 02:59:12.963873 12961 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.811s user 0.011s sys 0.022s
I20260321 02:59:12.907495 13135 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:12.946375 13203 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.890s user 0.010s sys 0.061s
I20260321 02:59:12.611910 13195 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:12.689393 13139 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:12.945672 13096 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.041s user 0.034s sys 0.040s
W20260321 02:59:12.959352 13143 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.780s user 0.004s sys 0.021s
I20260321 02:59:13.069537 11376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45634 (request call id 173) took 1260 ms. Trace:
W20260321 02:59:13.112957 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:13.117534 11376 rpcz_store.cc:276] 0321 02:59:11.809347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:11.824763 (+ 15416us) service_pool.cc:225] Handling call
0321 02:59:13.069356 (+1244593us) spinlock_profiling.cc:243] Waited 43.4 ms on lock 0x7b540030e654. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:13.069468 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":99789056}
I20260321 02:59:13.013815 13100 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:13.119969 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:13.120209 13100 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:13.120415 12961 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:13.121119 12961 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:13.124126 13235 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:13.124737 13235 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:13.126829 13235 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:13.099243 13238 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:13.128717 13238 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:13.133457 12846 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:13.135912 13189 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:13.011605 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:13.140847 13189 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:13.140518 13139 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:13.141355 13189 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:13.142731 13139 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:13.143355 13189 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:12.984700 12990 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:13.147225 11230 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:41224 (request call id 15) took 1257 ms. Trace:
W20260321 02:59:13.013999 13174 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.089s user 0.036s sys 0.032s
I20260321 02:59:13.093384 11309 raft_consensus.cc:2468] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:13.088826 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.097679 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.163340 11235 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:59:12.956833 13196 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:13.065538 13129 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.192s user 0.012s sys 0.031s
I20260321 02:59:12.960144 13123 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:12.948982 13084 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:13.133709 13181 log.cc:927] Time spent T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 1.721s user 0.006s sys 0.015s
I20260321 02:59:13.153764 11275 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:13.147653 11230 rpcz_store.cc:276] 0321 02:59:11.889381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:11.889540 (+ 159us) service_pool.cc:225] Handling call
0321 02:59:13.139697 (+1250157us) spinlock_profiling.cc:243] Waited 20.2 ms on lock 0x7b54001246a0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac4c9900 00007fa8ac546a32 00007fa8b1510cca 00007fa8aaa9d934 00007fa8aaa9d8c2 00007fa8aaa9d7f1 00007fa8aaa9d76c 00007fa8aaa9ca22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:13.147136 (+ 7439us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":46364032}
I20260321 02:59:13.070029 13097 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:13.164089 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:13.177592 11275 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:13.178120 11309 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:13.164573 11229 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
W20260321 02:59:13.168130 11379 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.741s user 0.013s sys 0.040s
W20260321 02:59:13.180788 11379 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.762s user 0.014s sys 0.040s
I20260321 02:59:13.092214 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.102293 13213 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:13.182668 11233 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:13.014849 13138 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:13.079144 13165 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:13.099357 13192 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:13.023167 13216 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:13.182441 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:13.159703 12961 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:13.211231 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.211035 11311 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:13.211186 11309 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:13.210997 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.206202 13174 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:59200 (request call id 8) took 12805 ms. Trace:
I20260321 02:59:13.214958 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:13.189126 11277 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.000s sys 0.032s
I20260321 02:59:13.216012 13123 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:13.217702 12924 consensus_meta.cc:287] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.880s user 0.012s sys 0.002s
W20260321 02:59:13.197227 11274 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.007s sys 0.007s
W20260321 02:59:13.225180 11274 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.535s user 0.007s sys 0.007s
I20260321 02:59:13.104139 13135 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
W20260321 02:59:13.223717 11277 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.559s user 0.000s sys 0.032s
I20260321 02:59:13.227561 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:13.228394 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:13.228678 11236 raft_consensus.cc:2410] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260321 02:59:13.231007 11350 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.566s user 0.065s sys 0.154s
I20260321 02:59:13.223719 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:13.255671 11311 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
W20260321 02:59:13.231429 11350 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.566s user 0.065s sys 0.154s
I20260321 02:59:13.175213 13196 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:13.382834 11942 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:13.383786 11309 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:13.282805 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:13.325915 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 698123us
I20260321 02:59:13.227459 12924 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:13.234328 11349 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.014s sys 0.036s
I20260321 02:59:13.258436 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:13.485594 11349 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.793s user 0.015s sys 0.058s
I20260321 02:59:13.485762 11303 raft_consensus.cc:1240] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:13.422366 11385 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1397) took 3094 ms. Trace:
W20260321 02:59:13.488610 11305 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.799s user 0.016s sys 0.020s
W20260321 02:59:13.537106 11305 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.847s user 0.016s sys 0.020s
I20260321 02:59:13.351501 11231 raft_consensus.cc:2468] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:13.328928 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.237376 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.540439 11233 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:13.491570 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:13.234145 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:13.510725 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:13.545790 11376 raft_consensus.cc:3060] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:13.437698 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:13.383746 13154 consensus_meta.cc:287] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 0.997s user 0.006s sys 0.005s
W20260321 02:59:13.514111 11942 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:13.555024 11942 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:13.258987 11977 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45078 (request call id 369) took 1655 ms. Trace:
I20260321 02:59:13.514310 13195 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:13.281410 13143 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:13.211548 13096 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:13.494733 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:13.566974 11311 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:59:13.487891 11385 rpcz_store.cc:276] 0321 02:59:10.327990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:10.328114 (+ 124us) service_pool.cc:225] Handling call
0321 02:59:10.356900 (+ 28786us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:10.587635 (+230735us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:10.590748 (+ 3113us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:10.604017 (+ 13269us) log.cc:844] Serialized 26 byte log entry
0321 02:59:10.770037 (+166020us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:59:10.770261 (+ 224us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:11.909281 (+1139020us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b540009ae54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:11.909372 (+ 91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:13.133366 (+1223994us) spinlock_profiling.cc:243] Waited 32.6 ms on lock 0x7b24002665b8. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93088f4 00007fa8a93069fb 00007fa8a9395799 00007fa8a9395425 00007fa8a9394a6d 00007fa8ac0cf4ae 00007fa8ac0e6c2c 00007fa8ac0e5f56 00007fa8ac0e5b78 00007fa8ac0ef5c1
0321 02:59:13.422113 (+288747us) raft_consensus.cc:1685] finished
0321 02:59:13.422144 (+ 31us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:13.422303 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4305,"raft.queue_time_us":146,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":16054,"spinlock_wait_cycles":591099520,"thread_start_us":108058,"threads_started":1,"wal-append.queue_time_us":786544}
I20260321 02:59:13.233520 13203 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:13.577519 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.578083 11376 raft_consensus.cc:2468] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:13.234181 13238 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:13.280697 13093 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:13.582686 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:13.583556 11311 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
W20260321 02:59:13.527187 13086 consensus_meta.cc:287] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.909s user 0.007s sys 0.002s
I20260321 02:59:13.531339 13242 raft_consensus.cc:2804] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:13.507021 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.458024 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:13.541741 11309 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
W20260321 02:59:13.547341 11307 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.031s sys 0.025s
W20260321 02:59:13.446441 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:13.587786 11307 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.687s user 0.032s sys 0.025s
I20260321 02:59:13.587024 11306 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
W20260321 02:59:13.588325 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:13.588053 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:13.532733 11943 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.838s user 0.016s sys 0.013s
I20260321 02:59:13.550734 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:13.483587 13133 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.855s user 0.001s sys 0.016s
I20260321 02:59:13.421854 13123 raft_consensus.cc:2804] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:13.556473 11349 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:13.596239 11349 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:13.590731 13133 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.979s user 0.002s sys 0.016s
I20260321 02:59:13.603706 12961 consensus_queue.cc:1048] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:13.605707 12961 raft_consensus.cc:2804] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:13.606240 12961 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:13.606678 12961 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:13.589563 13129 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:13.608623 11308 raft_consensus.cc:1240] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:13.608498 13129 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:13.611234 13129 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:13.558043 11976 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:59:13.558156 13168 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:13.571033 12924 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:13.570899 13143 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:13.545255 12880 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:13.574347 13210 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:13.595149 13123 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:13.578835 11942 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:13.580287 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.543140 13198 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
W20260321 02:59:13.589169 11943 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.265s user 0.017s sys 0.014s
I20260321 02:59:13.589937 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:13.590960 11310 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s user 0.005s sys 0.010s
W20260321 02:59:13.649655 11310 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.749s user 0.005s sys 0.010s
I20260321 02:59:13.567536 13154 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:13.650345 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 71706.3us
I20260321 02:59:13.592017 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:13.548659 13084 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:13.576339 13172 raft_consensus.cc:2804] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:13.619478 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:13.558156 11977 rpcz_store.cc:276] 0321 02:59:11.603311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:11.605457 (+ 2146us) service_pool.cc:225] Handling call
0321 02:59:11.624248 (+ 18791us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:11.783881 (+159633us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:11.783922 (+ 41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:11.784000 (+ 78us) raft_consensus.cc:1645] Marking committed up to 6
0321 02:59:11.784044 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:13.258724 (+1474680us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b54005d0a54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:13.258780 (+ 56us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:13.258928 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":356519808}
I20260321 02:59:13.621999 11378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40950 (request call id 1065) took 3139 ms. Trace:
I20260321 02:59:13.590471 13093 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:13.624155 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.624795 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.686985 11231 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:13.597860 13086 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:13.231220 13208 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.699s user 0.012s sys 0.006s
I20260321 02:59:13.587286 13261 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:13.585433 13242 raft_consensus.cc:697] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 4 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:13.642200 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.590031 13238 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:13.594707 13185 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:13.519093 12990 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:13.576324 13203 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:13.644263 11201 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:13.585783 13096 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:13.692394 13208 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.185s user 0.016s sys 0.006s
I20260321 02:59:13.686550 13172 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:13.721660 11304 raft_consensus.cc:1240] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 2. Current term is 3. Ops: []
W20260321 02:59:13.728143 11379 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.018s sys 0.021s
W20260321 02:59:13.730436 11379 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.523s user 0.018s sys 0.021s
I20260321 02:59:13.731976 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
W20260321 02:59:13.651764 11940 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.017s sys 0.012s
W20260321 02:59:13.740612 11940 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.634s user 0.017s sys 0.012s
I20260321 02:59:13.731632 13093 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:13.645679 12924 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:13.686558 11229 raft_consensus.cc:2468] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:59:13.683790 13195 raft_consensus.cc:2804] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:13.648375 13123 raft_consensus.cc:3060] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:13.225472 13174 rpcz_store.cc:276] 0321 02:59:00.416421 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:00.422442 (+ 6021us) service_pool.cc:225] Handling call
0321 02:59:00.477335 (+ 54893us) log.cc:844] Serialized 431 byte log entry
0321 02:59:13.184898 (+12707563us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":13325,"raft.run_cpu_time_us":967,"raft.run_wall_time_us":1012,"thread_start_us":36770,"threads_started":2,"wal-append.queue_time_us":74444,"wal-append.run_cpu_time_us":3585,"wal-append.run_wall_time_us":1074939}
W20260321 02:59:13.701288 11232 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.944s user 0.003s sys 0.035s
W20260321 02:59:13.701884 11351 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.845s user 0.013s sys 0.019s
W20260321 02:59:13.764199 11232 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.007s user 0.004s sys 0.035s
W20260321 02:59:13.764353 11351 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.976s user 0.014s sys 0.020s
I20260321 02:59:13.728358 13134 raft_consensus.cc:2804] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:59:13.647372 13216 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:13.705911 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.808219 13134 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:13.808674 11236 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:13.809793 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.644627 12880 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
W20260321 02:59:13.711359 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 67181.1us
I20260321 02:59:13.681959 13097 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:13.682096 11942 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:13.680485 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:13.648540 13210 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:13.745955 11302 raft_consensus.cc:2393] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:13.747922 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:13.687837 13128 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.011s sys 0.021s
I20260321 02:59:13.773399 11304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1425) took 1158 ms. Trace:
W20260321 02:59:13.774528 11972 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45098 (request call id 302) took 2109 ms (client timeout 2180 ms). Trace:
I20260321 02:59:13.777371 11303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1422) took 1163 ms. Trace:
I20260321 02:59:13.703895 13261 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:13.810120 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.824458 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:13.824673 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 142696us
I20260321 02:59:13.825057 11231 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:13.810964 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.826529 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:13.827207 11376 raft_consensus.cc:2468] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 1.
I20260321 02:59:13.818712 11234 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58200 (request call id 1481) took 1428 ms. Trace:
I20260321 02:59:13.829795 11304 rpcz_store.cc:276] 0321 02:59:12.614659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:13.606097 (+991438us) service_pool.cc:225] Handling call
0321 02:59:13.606442 (+ 345us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:13.730273 (+123831us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:13.773029 (+ 42756us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b54001adbd4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:13.773309 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":264379648}
I20260321 02:59:13.790274 13213 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:13.831544 11303 rpcz_store.cc:276] 0321 02:59:12.613864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:12.615239 (+ 1375us) service_pool.cc:225] Handling call
0321 02:59:12.615485 (+ 246us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:13.486392 (+870907us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:13.776948 (+290556us) spinlock_profiling.cc:243] Waited 870 ms on lock 0x7b54001accd4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:13.777168 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2000977152}
I20260321 02:59:13.833420 11230 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:13.739723 11379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40950 (request call id 1067) took 2544 ms. Trace:
I20260321 02:59:13.834436 11379 rpcz_store.cc:276] 0321 02:59:11.195836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:12.173238 (+977402us) service_pool.cc:225] Handling call
0321 02:59:12.173698 (+ 460us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:12.174586 (+ 888us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:12.174616 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:12.184192 (+ 9576us) log.cc:844] Serialized 26 byte log entry
0321 02:59:13.181359 (+997167us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:13.730866 (+549507us) log.cc:844] Serialized 14 byte log entry
0321 02:59:13.739279 (+ 8413us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:13.739388 (+ 109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:13.739448 (+ 60us) raft_consensus.cc:1685] finished
0321 02:59:13.739466 (+ 18us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:13.739647 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":303,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":274,"thread_start_us":1287057,"threads_started":2,"wal-append.queue_time_us":1208682}
I20260321 02:59:13.797039 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:13.800890 11381 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1409) took 3403 ms. Trace:
I20260321 02:59:13.639268 13168 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:13.734063 13172 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:13.809954 13203 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:13.795080 13248 log.cc:927] Time spent T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.565s user 0.002s sys 0.000s
I20260321 02:59:13.917637 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.930883 11233 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:13.920416 13086 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:13.935251 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:13.761493 13195 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:13.829072 11234 rpcz_store.cc:276] 0321 02:59:12.390500 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:12.395705 (+ 5205us) service_pool.cc:225] Handling call
0321 02:59:12.587211 (+191506us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:13.103894 (+516683us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:13.103941 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:13.104029 (+ 88us) raft_consensus.cc:1645] Marking committed up to 3
0321 02:59:13.104074 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:13.818378 (+714304us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b5400333bd4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:13.818443 (+ 65us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:13.818634 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1187609984}
I20260321 02:59:13.686410 11378 rpcz_store.cc:276] 0321 02:59:10.482796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:12.003297 (+1520501us) service_pool.cc:225] Handling call
0321 02:59:12.198346 (+195049us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:12.386555 (+188209us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:12.406312 (+ 19757us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:12.419677 (+ 13365us) raft_consensus.cc:1645] Marking committed up to 5
0321 02:59:12.427763 (+ 8086us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:12.508281 (+ 80518us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:13.621682 (+1113401us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b54002c0450. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54bc8d 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:13.621904 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":417284352}
W20260321 02:59:13.922191 11228 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58214 (request call id 1097) took 2203 ms (client timeout 2813 ms). Trace:
I20260321 02:59:13.749244 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:13.711443 13238 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:13.945523 11228 rpcz_store.cc:269] 0321 02:59:11.633793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:11.633958 (+ 165us) service_pool.cc:225] Handling call
0321 02:59:13.836663 (+2202705us) spinlock_profiling.cc:243] Waited 28.6 ms on lock 0x7b540019d2d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:13.836793 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":65852672}
I20260321 02:59:13.919175 13194 raft_consensus.cc:2804] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:13.923069 11304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56616 (request call id 1538) took 1119 ms. Trace:
W20260321 02:59:13.842106 11276 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.9.213.4:41285 timed out after 1.655s (SENT) [suppressed 11 similar messages]
W20260321 02:59:13.923692 11274 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.668s user 0.019s sys 0.019s
I20260321 02:59:13.720927 13262 raft_consensus.cc:2804] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election won for term 4
W20260321 02:59:13.822789 13255 log.cc:927] Time spent T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260321 02:59:13.929672 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.951875 11236 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:13.923331 13139 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:13.934654 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.939431 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.956995 11231 raft_consensus.cc:2410] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260321 02:59:13.939474 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:13.940824 11233 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:13.958329 11233 raft_consensus.cc:606] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:13.943833 11978 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:13.837386 13130 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:13.808709 13134 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:13.825997 13196 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:13.820358 12990 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:13.822383 13096 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:13.832974 13128 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.728s user 0.014s sys 0.021s
I20260321 02:59:13.948817 11304 rpcz_store.cc:276] 0321 02:59:12.803365 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:13.831434 (+1028069us) service_pool.cc:225] Handling call
0321 02:59:13.922445 (+ 91011us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:13.922654 (+ 209us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:13.922691 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:13.922746 (+ 55us) raft_consensus.cc:1645] Marking committed up to 3
0321 02:59:13.922789 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:13.922866 (+ 77us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:13.923001 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260321 02:59:13.949880 11274 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.694s user 0.019s sys 0.019s
I20260321 02:59:13.921155 11381 rpcz_store.cc:276] 0321 02:59:10.397915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:10.917590 (+519675us) service_pool.cc:225] Handling call
0321 02:59:10.917938 (+ 348us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:10.918325 (+ 387us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:10.918354 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:12.040168 (+1121814us) spinlock_profiling.cc:243] Waited 3.49 ms on lock 0x7b2c000d0e18. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8aae48c81 00007fa8aae585e2 00007fa8ac587de1 00007fa8ac54c81a 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22
0321 02:59:12.042429 (+ 2261us) log.cc:844] Serialized 26 byte log entry
0321 02:59:12.931050 (+888621us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:59:12.950669 (+ 19619us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:12.950922 (+ 253us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:13.800329 (+849407us) spinlock_profiling.cc:243] Waited 31.9 ms on lock 0x7b540015d720. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac4c219e 00007fa8ac4ccd10 00007fa8ac4cccb8 00007fa8ac4ccc38 00007fa8ac4ccbf3 00007fa8ac4cbe7c 00007fa8b238ff06 00007fa8b238a6b8 00007fa8ac508bab 00007fa8ac50b6a0
0321 02:59:13.800649 (+ 320us) raft_consensus.cc:1685] finished
0321 02:59:13.800668 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:13.800825 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":385,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":118,"spinlock_wait_cycles":81465856,"thread_start_us":886034,"threads_started":1,"wal-append.queue_time_us":1339315}
I20260321 02:59:13.929989 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:14.037011 11376 raft_consensus.cc:3060] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:13.925420 13229 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:13.760425 12924 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:13.823455 13154 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:13.964068 13261 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:13.966006 13264 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:13.968247 13168 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:13.972088 11944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:13.972569 13226 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:13.977478 12880 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:13.986310 11182 auto_rebalancer.cc:799] Could not move replica: Incomplete: tablet 404c3f80af814c3fa8db2501818b1e33, TS fef7f8cc55b043039131bd9b2229f0a5 -> TS 3d8d3176d6ec407ea2a965556c2e2c56 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:13.829578 11972 rpcz_store.cc:269] 0321 02:59:11.664875 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:11.666912 (+ 2037us) service_pool.cc:225] Handling call
0321 02:59:13.774308 (+2107396us) spinlock_profiling.cc:243] Waited 4.8 ms on lock 0x7b54006400d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:13.774425 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11040000}
I20260321 02:59:13.993084 11132 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42502 (request call id 224) took 4754 ms. Trace:
I20260321 02:59:13.940192 13143 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:14.074966 11182 auto_rebalancer.cc:284] scheduled replica move failed to complete: Incomplete: tablet 404c3f80af814c3fa8db2501818b1e33, TS fef7f8cc55b043039131bd9b2229f0a5 -> TS 3d8d3176d6ec407ea2a965556c2e2c56 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:13.994503 11275 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
W20260321 02:59:14.084914 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 90482.7us
W20260321 02:59:14.087594 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:14.087832 13172 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:14.088371 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:14.088582 13216 raft_consensus.cc:2804] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:14.089174 13216 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:14.089624 13216 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:14.090181 13172 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:13.958365 13260 raft_consensus.cc:493] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:13.946033 13195 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:14.068506 13264 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:13.948185 13165 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:14.041092 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:14.120379 11233 raft_consensus.cc:2468] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:13.820521 13242 consensus_queue.cc:237] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.080821 11234 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:14.132292 13238 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:14.135278 13267 log.cc:927] Time spent T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.121s user 0.000s sys 0.001s
I20260321 02:59:14.135540 13234 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:14.135989 13258 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:14.136488 13258 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.139438 13258 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:14.141244 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:14.011546 11236 raft_consensus.cc:1275] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260321 02:59:14.096180 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:14.080732 11132 rpcz_store.cc:276] 0321 02:59:09.238979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:09.239276 (+ 297us) service_pool.cc:225] Handling call
0321 02:59:13.992992 (+4753716us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:59:09.961136 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:59:10.072166 (+111030us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:59:10.074611 (+ 2445us) write_op.cc:435] Acquired schema lock
0321 02:59:10.074634 (+ 23us) tablet.cc:661] Decoding operations
0321 02:59:10.185279 (+110645us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:59:10.185341 (+ 62us) write_op.cc:641] Partition lock acquired for write op
0321 02:59:10.185371 (+ 30us) tablet.cc:684] Acquiring locks for 9 operations
0321 02:59:10.186165 (+ 794us) tablet.cc:700] Row locks acquired
0321 02:59:10.186186 (+ 21us) write_op.cc:260] PREPARE: finished
0321 02:59:10.186333 (+ 147us) write_op.cc:270] Start()
0321 02:59:10.186481 (+ 148us) write_op.cc:276] Timestamp: P: 1774061950186287 usec, L: 0
0321 02:59:10.186506 (+ 25us) op_driver.cc:348] REPLICATION: starting
0321 02:59:10.187158 (+ 652us) log.cc:844] Serialized 3764 byte log entry
0321 02:59:10.512500 (+325342us) op_driver.cc:464] REPLICATION: finished
0321 02:59:11.595578 (+1083078us) write_op.cc:301] APPLY: starting
0321 02:59:11.602380 (+ 6802us) tablet.cc:1366] starting BulkCheckPresence
0321 02:59:11.687034 (+ 84654us) tablet.cc:1369] finished BulkCheckPresence
0321 02:59:11.720424 (+ 33390us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:59:11.893366 (+172942us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:59:11.893598 (+ 232us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0321 02:59:11.893641 (+ 43us) write_op.cc:312] APPLY: finished
0321 02:59:11.893943 (+ 302us) log.cc:844] Serialized 71 byte log entry
0321 02:59:12.395443 (+501500us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:12.424519 (+ 29076us) write_op.cc:454] Released schema lock
0321 02:59:12.453683 (+ 29164us) write_op.cc:341] FINISH: Updating metrics
0321 02:59:12.704377 (+250694us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:12.756951 (+ 52574us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":59215,"reported_tablets":29,"child_traces":[["op",{"apply.queue_time_us":1038189,"apply.run_cpu_time_us":25410,"apply.run_wall_time_us":1025080,"num_ops":9,"prepare.queue_time_us":1817,"prepare.run_cpu_time_us":5700,"prepare.run_wall_time_us":230097,"replication_time_us":263748,"thread_start_us":394739,"threads_started":2,"wal-append.queue_time_us":843117,"wal-append.run_cpu_time_us":1560,"wal-append.run_wall_time_us":1103772}]]}
I20260321 02:59:14.013931 13262 raft_consensus.cc:697] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:14.047340 13220 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:14.100178 11376 raft_consensus.cc:2468] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 6.
I20260321 02:59:14.149642 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:14.018617 13263 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:13.993855 13194 raft_consensus.cc:697] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:14.113933 11302 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56630 (request call id 1429) took 1498 ms. Trace:
I20260321 02:59:13.949891 13086 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:14.047036 13269 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:14.068172 12976 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:14.115340 11201 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:14.117169 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:14.122252 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:14.125834 12924 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:14.126096 13264 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:13.995088 13139 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.126922 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:14.200703 11231 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
W20260321 02:59:14.128731 13268 log.cc:927] Time spent T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.055s user 0.000s sys 0.001s
I20260321 02:59:14.129726 13210 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:14.130088 12720 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:14.132375 11234 raft_consensus.cc:606] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:14.154506 13220 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:14.210623 11234 raft_consensus.cc:614] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260321 02:59:14.097044 11227 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.680s user 0.021s sys 0.046s
W20260321 02:59:14.097268 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 501087us
I20260321 02:59:14.145219 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:14.163872 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:14.221906 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.165493 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:14.167863 13138 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:14.174328 11350 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.687s user 0.011s sys 0.071s
I20260321 02:59:14.095824 13226 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.176677 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.224472 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.222581 11381 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:14.225112 11376 raft_consensus.cc:2468] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:14.226820 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:14.227428 11231 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:14.228044 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.228781 11385 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:14.218111 11974 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
W20260321 02:59:14.232220 11305 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.032s sys 0.075s
I20260321 02:59:14.198513 13086 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.178086 11302 rpcz_store.cc:276] 0321 02:59:12.615828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:12.620881 (+ 5053us) service_pool.cc:225] Handling call
0321 02:59:14.113726 (+1492845us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54003c82d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:14.113876 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2584235264}
I20260321 02:59:14.117126 13260 raft_consensus.cc:515] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:14.196481 11304 raft_consensus.cc:3060] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:14.205909 13135 consensus_meta.cc:287] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.943s user 0.010s sys 0.000s
I20260321 02:59:14.207096 13084 raft_consensus.cc:2804] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260321 02:59:14.200999 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 85743.5us
I20260321 02:59:14.207252 11134 catalog_manager.cc:5671] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 3 to 5, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 5 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
W20260321 02:59:14.212869 11311 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.030s sys 0.043s
W20260321 02:59:14.238600 11311 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.552s user 0.032s sys 0.045s
I20260321 02:59:14.237017 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.239328 11134 catalog_manager.cc:5671] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 2, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 2 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
W20260321 02:59:14.228061 11310 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.055s sys 0.079s
W20260321 02:59:14.241271 11310 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.555s user 0.055s sys 0.079s
I20260321 02:59:14.209461 11200 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:14.223591 11350 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.736s user 0.012s sys 0.071s
I20260321 02:59:14.174535 13076 consensus_queue.cc:1048] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
W20260321 02:59:14.241889 11350 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 759670us
W20260321 02:59:14.244027 11974 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45088 (request call id 238) took 2351 ms (client timeout 1739 ms). Trace:
W20260321 02:59:14.244634 11974 rpcz_store.cc:269] 0321 02:59:11.892536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:11.892688 (+ 152us) service_pool.cc:225] Handling call
0321 02:59:14.243815 (+2351127us) spinlock_profiling.cc:243] Waited 2.31 s on lock 0x7b54004df8d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:14.243935 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5311892864}
I20260321 02:59:14.246721 13139 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:14.246918 11972 raft_consensus.cc:1240] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:14.248814 13226 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:14.251528 13011 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:14.196632 13263 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.252108 13011 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:14.253777 13263 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:14.198714 13269 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.254469 13011 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:14.255224 11973 raft_consensus.cc:2468] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:14.255429 13260 leader_election.cc:290] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:14.256043 13269 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:14.267304 11308 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.000s sys 0.015s
I20260321 02:59:14.244107 13138 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:14.267709 11308 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.505s user 0.000s sys 0.015s
I20260321 02:59:14.247980 13084 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
W20260321 02:59:14.256880 11973 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45098 (request call id 303) took 2584 ms (client timeout 1655 ms). Trace:
I20260321 02:59:14.269486 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.258103 11302 raft_consensus.cc:1275] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260321 02:59:14.258525 13216 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.260965 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.262351 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.304603 11308 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1428) took 1689 ms. Trace:
I20260321 02:59:14.191107 11275 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:14.306674 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 115662us
I20260321 02:59:14.309036 11275 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:14.263911 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.265355 11310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56616 (request call id 1527) took 1732 ms. Trace:
W20260321 02:59:14.255736 12961 consensus_meta.cc:287] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.648s user 0.007s sys 0.002s
I20260321 02:59:14.310427 11379 raft_consensus.cc:2393] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 4: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
W20260321 02:59:14.229514 11227 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.830s user 0.022s sys 0.046s
W20260321 02:59:14.267035 11182 auto_rebalancer.cc:791] Removing replace marker failed: Replica 0d7efc590a1041a9bd4cd4a971332dfb is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
W20260321 02:59:14.314628 11182 auto_rebalancer.cc:799] Could not move replica: Incomplete: tablet 446eeb6a12c64deb848b6db18a6587f6, TS 0d7efc590a1041a9bd4cd4a971332dfb -> TS 3d8d3176d6ec407ea2a965556c2e2c56 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.256121 13135 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.271559 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:14.315615 11307 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.056s sys 0.102s
W20260321 02:59:14.316504 11307 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.607s user 0.056s sys 0.102s
I20260321 02:59:14.316533 11384 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:14.260349 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.309551 13216 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:14.310101 11236 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:14.330278 11236 raft_consensus.cc:606] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:14.266331 11232 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58200 (request call id 1480) took 1882 ms. Trace:
I20260321 02:59:14.310887 11381 raft_consensus.cc:2393] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 3: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:14.312027 11385 raft_consensus.cc:2393] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 3: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:14.305140 11308 rpcz_store.cc:276] 0321 02:59:12.615472 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:13.611084 (+995612us) service_pool.cc:225] Handling call
0321 02:59:13.611377 (+ 293us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:13.611732 (+ 355us) raft_consensus.cc:1534] Early marking committed up to index 3
0321 02:59:13.762881 (+151149us) log.cc:844] Serialized 14 byte log entry
0321 02:59:14.304207 (+541326us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:14.304264 (+ 57us) raft_consensus.cc:1645] Marking committed up to 3
0321 02:59:14.304310 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:14.304393 (+ 83us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:14.304538 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35495,"prepare.queue_time_us":598566,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"thread_start_us":626976,"threads_started":2}
I20260321 02:59:14.313416 11275 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:14.331207 11232 rpcz_store.cc:276] 0321 02:59:12.384001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:12.384196 (+ 195us) service_pool.cc:225] Handling call
0321 02:59:12.384685 (+ 489us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:12.641774 (+257089us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:12.641823 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:12.665369 (+ 23546us) log.cc:844] Serialized 26 byte log entry
0321 02:59:13.764893 (+1099524us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:13.764956 (+ 63us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:14.142120 (+377164us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b540037a354. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:14.142163 (+ 43us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:14.262937 (+120774us) raft_consensus.cc:1685] finished
0321 02:59:14.262965 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:14.266260 (+ 3295us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":51399,"raft.run_cpu_time_us":2695,"raft.run_wall_time_us":62774,"spinlock_wait_cycles":590013312,"thread_start_us":1007104,"threads_started":1,"wal-append.queue_time_us":1236900}
I20260321 02:59:14.310174 11310 rpcz_store.cc:276] 0321 02:59:12.533290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:12.533443 (+ 153us) service_pool.cc:225] Handling call
0321 02:59:12.612659 (+ 79216us) raft_consensus.cc:1471] Updating replica for 2 ops
0321 02:59:12.689414 (+ 76755us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:12.689457 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0321 02:59:12.690189 (+ 732us) log.cc:844] Serialized 52 byte log entry
0321 02:59:13.650045 (+959856us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:14.241666 (+591621us) log.cc:844] Serialized 14 byte log entry
0321 02:59:14.265008 (+ 23342us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:14.265095 (+ 87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:14.265140 (+ 45us) raft_consensus.cc:1685] finished
0321 02:59:14.265152 (+ 12us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:14.265297 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22507,"prepare.queue_time_us":560529,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":509943,"raft.run_cpu_time_us":543,"raft.run_wall_time_us":44428,"thread_start_us":1304658,"threads_started":2,"wal-append.queue_time_us":1004587}
W20260321 02:59:14.232684 11305 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s user 0.032s sys 0.075s
I20260321 02:59:14.217694 13123 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:14.295249 11304 raft_consensus.cc:2468] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
W20260321 02:59:14.269915 11973 rpcz_store.cc:269] 0321 02:59:11.672227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:11.676073 (+ 3846us) service_pool.cc:225] Handling call
0321 02:59:14.256501 (+2580428us) spinlock_profiling.cc:243] Waited 2.52 s on lock 0x7b54004222d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:14.256807 (+ 306us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5802151808}
I20260321 02:59:14.273149 13084 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:14.326479 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.265136 13235 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
W20260321 02:59:14.315101 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 486091us
I20260321 02:59:14.333532 11275 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:14.343182 13272 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:14.343706 13272 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:14.344187 13272 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:14.345301 11275 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:14.347178 13210 raft_consensus.cc:2804] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:14.347847 13210 raft_consensus.cc:697] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
W20260321 02:59:14.314952 11182 auto_rebalancer.cc:284] scheduled replica move failed to complete: Incomplete: tablet 446eeb6a12c64deb848b6db18a6587f6, TS 0d7efc590a1041a9bd4cd4a971332dfb -> TS 3d8d3176d6ec407ea2a965556c2e2c56 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.348094 13276 raft_consensus.cc:2804] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:14.348913 13276 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:14.349383 13276 raft_consensus.cc:3060] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:14.235031 13262 consensus_queue.cc:237] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.350330 13135 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:14.338352 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.353233 11379 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:14.354619 11306 raft_consensus.cc:2393] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 6: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:14.354929 11305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1423) took 1740 ms. Trace:
I20260321 02:59:14.355424 13096 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:14.355774 11306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56616 (request call id 1544) took 1418 ms. Trace:
I20260321 02:59:14.359088 11306 rpcz_store.cc:276] 0321 02:59:12.937747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:13.588674 (+650927us) service_pool.cc:225] Handling call
0321 02:59:14.355545 (+766871us) spinlock_profiling.cc:243] Waited 670 ms on lock 0x7b54001d56d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:14.355685 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1540433152}
I20260321 02:59:14.355301 11305 rpcz_store.cc:276] 0321 02:59:12.614123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:12.614673 (+ 550us) service_pool.cc:225] Handling call
0321 02:59:12.614968 (+ 295us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:13.564033 (+949065us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:13.564082 (+ 49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:13.598276 (+ 34194us) log.cc:844] Serialized 26 byte log entry
0321 02:59:14.334264 (+735988us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:59:14.334301 (+ 37us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:14.334364 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:14.354719 (+ 20355us) raft_consensus.cc:1685] finished
0321 02:59:14.354740 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:14.354876 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49437,"raft.queue_time_us":512079,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":111,"thread_start_us":1538565,"threads_started":2,"wal-append.queue_time_us":716374}
I20260321 02:59:14.316332 13260 raft_consensus.cc:2804] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:14.316488 13226 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:14.386230 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.390967 11381 raft_consensus.cc:2393] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 6: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
W20260321 02:59:14.371799 13266 log.cc:927] Time spent T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20260321 02:59:14.369131 13210 consensus_queue.cc:237] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.305102 13138 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.356081 13096 raft_consensus.cc:697] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:14.315924 12961 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.382321 13260 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:14.270557 13194 consensus_queue.cc:237] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:14.415575 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.394979 13226 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:14.383406 13220 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:14.318293 13203 consensus_queue.cc:1059] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.076s
I20260321 02:59:14.401612 11227 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58200 (request call id 1496) took 1291 ms. Trace:
I20260321 02:59:14.422701 11227 rpcz_store.cc:276] 0321 02:59:13.110284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:13.149903 (+ 39619us) service_pool.cc:225] Handling call
0321 02:59:13.218562 (+ 68659us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:13.259350 (+ 40788us) raft_consensus.cc:1534] Early marking committed up to index 3
0321 02:59:13.324629 (+ 65279us) log.cc:844] Serialized 14 byte log entry
0321 02:59:13.513673 (+189044us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x7b1800310c60. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b2300441 00007fa8b230cca1 00007fa8a94b62f2 00007fa8a94b8591 00007fa8a94b8549 00007fa8a94b84d9 00007fa8a94b84a1 00007fa8a94b779d 0000000000379254 0000000000379099 00007fa8a949974a
0321 02:59:14.339015 (+825342us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:14.360439 (+ 21424us) raft_consensus.cc:1645] Marking committed up to 3
0321 02:59:14.375670 (+ 15231us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:14.387887 (+ 12217us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:14.400661 (+ 12774us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":31760,"prepare.run_cpu_time_us":590,"prepare.run_wall_time_us":3718,"raft.queue_time_us":37252,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"spinlock_wait_cycles":36653568,"thread_start_us":928195,"threads_started":1,"wal-append.queue_time_us":988088}
I20260321 02:59:14.423980 13195 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:14.425848 13138 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:14.426013 12961 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:14.426621 11980 raft_consensus.cc:2368] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4.
I20260321 02:59:14.376609 13235 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.427313 13203 raft_consensus.cc:3055] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:14.416316 11381 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:14.428678 13086 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:14.415194 13260 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:14.420815 13226 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:14.412247 11378 raft_consensus.cc:2410] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260321 02:59:14.439934 11182 auto_rebalancer.cc:791] Removing replace marker failed: Replica 83b12b883a7f4129a87dadf5a78f08da is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
W20260321 02:59:14.440582 11182 auto_rebalancer.cc:799] Could not move replica: Incomplete: tablet f9e8870ff9214a65af6cd9f677bffab4, TS 83b12b883a7f4129a87dadf5a78f08da -> TS 3d8d3176d6ec407ea2a965556c2e2c56 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:14.440963 11182 auto_rebalancer.cc:284] scheduled replica move failed to complete: Incomplete: tablet f9e8870ff9214a65af6cd9f677bffab4, TS 83b12b883a7f4129a87dadf5a78f08da -> TS 3d8d3176d6ec407ea2a965556c2e2c56 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.428537 13176 raft_consensus.cc:993] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb: : Instructing follower fef7f8cc55b043039131bd9b2229f0a5 to start an election
I20260321 02:59:14.427690 13203 raft_consensus.cc:740] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:14.426239 13195 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:14.368155 11942 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:14.455726 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 87683.7us
I20260321 02:59:14.456861 11942 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:14.457228 11311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56616 (request call id 1537) took 1668 ms. Trace:
I20260321 02:59:14.459192 12976 raft_consensus.cc:2804] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:14.464557 12976 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:14.410403 13276 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.464958 12976 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:14.472534 13273 log.cc:927] Time spent T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20260321 02:59:14.474260 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.475301 11378 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
W20260321 02:59:14.458694 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 616569us
I20260321 02:59:14.459532 13192 raft_consensus.cc:1081] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Signalling peer fef7f8cc55b043039131bd9b2229f0a5 to start an election
I20260321 02:59:14.457746 11311 rpcz_store.cc:276] 0321 02:59:12.789249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:13.682730 (+893481us) service_pool.cc:225] Handling call
0321 02:59:13.683300 (+ 570us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:13.683565 (+ 265us) raft_consensus.cc:1534] Early marking committed up to index 3
0321 02:59:14.239009 (+555444us) log.cc:844] Serialized 14 byte log entry
0321 02:59:14.456630 (+217621us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:14.456702 (+ 72us) raft_consensus.cc:1645] Marking committed up to 3
0321 02:59:14.456899 (+ 197us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:14.456995 (+ 96us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:14.457158 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3126,"prepare.queue_time_us":527178,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":374,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":5722,"thread_start_us":768662,"threads_started":2}
I20260321 02:59:14.461498 13220 raft_consensus.cc:2804] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:14.506690 13134 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:14.507161 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.507999 11379 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:14.508320 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:14.508929 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:14.509104 11376 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:14.509582 11380 raft_consensus.cc:3060] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:14.509690 11228 raft_consensus.cc:1240] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:14.511313 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.516211 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:14.516726 11380 raft_consensus.cc:2468] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:14.520246 13188 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:14.520772 11971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:14.520823 13188 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.508322 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.524600 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:14.525336 11381 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:14.521399 11974 raft_consensus.cc:1240] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:14.493158 13220 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:14.521274 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:14.538787 11276 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:14.544600 11276 leader_election.cc:400] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Vote denied by peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285) with higher term. Message: Invalid argument: T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4.
I20260321 02:59:14.545042 11276 leader_election.cc:403] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260321 02:59:14.548624 11235 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:59:14.521611 11971 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:14.518999 13134 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:14.458746 11942 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:14.556941 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 98294us
I20260321 02:59:14.557793 13086 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:14.557808 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:14.476179 13096 consensus_queue.cc:237] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.543475 13220 raft_consensus.cc:3060] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:14.563869 11276 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:14.550201 13264 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:14.549947 13263 raft_consensus.cc:2749] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:14.566951 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.552691 13123 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:14.575220 11381 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:14.497340 13203 consensus_queue.cc:260] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:14.571458 11376 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:14.584887 13272 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.568908 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:14.562919 12976 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:14.579105 11307 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1420) took 1966 ms. Trace:
I20260321 02:59:14.580566 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.589702 11382 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:14.589347 11307 rpcz_store.cc:276] 0321 02:59:12.613218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:12.624281 (+ 11063us) service_pool.cc:225] Handling call
0321 02:59:12.624743 (+ 462us) raft_consensus.cc:1471] Updating replica for 2 ops
0321 02:59:13.588229 (+963486us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:13.588266 (+ 37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0321 02:59:13.650035 (+ 61769us) log.cc:844] Serialized 52 byte log entry
0321 02:59:14.317236 (+667201us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:14.357700 (+ 40464us) log.cc:844] Serialized 14 byte log entry
0321 02:59:14.357941 (+ 241us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:14.358035 (+ 94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:14.432305 (+ 74270us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b4800061a58. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b14721a1 00007fa8b1469237 00007fa8b14690ca 00007fa8b156092f 00007fa8b15605f7 00007fa8b1574c1c 00007fa8b1574bc8 00007fa8b1574b48 00007fa8b1574b03 00007fa8b1573dac 00007fa8b04acdf6
0321 02:59:14.578709 (+146404us) raft_consensus.cc:1685] finished
0321 02:59:14.578818 (+ 109us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:14.579025 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65547,"prepare.queue_time_us":40743,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":518859,"raft.run_cpu_time_us":3107,"raft.run_wall_time_us":73981,"spinlock_wait_cycles":37955456,"thread_start_us":1295385,"threads_started":2,"wal-append.queue_time_us":838883}
I20260321 02:59:14.550268 13226 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.562427 13195 raft_consensus.cc:2749] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:14.584096 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.584874 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:14.565420 13135 raft_consensus.cc:2804] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:14.592229 11971 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:14.582913 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.600208 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.565238 13216 consensus_queue.cc:1059] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260321 02:59:14.602108 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.602317 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:14.612645 11380 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:14.603750 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:14.613739 11382 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:14.605204 11228 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58214 (request call id 1111) took 1371 ms. Trace:
I20260321 02:59:14.603866 12976 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:14.570003 13086 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.603318 13272 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:14.630896 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 421535us
I20260321 02:59:14.614929 11228 rpcz_store.cc:276] 0321 02:59:13.233415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:13.957002 (+723587us) service_pool.cc:225] Handling call
0321 02:59:14.017506 (+ 60504us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:14.510081 (+492575us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:14.604912 (+ 94831us) spinlock_profiling.cc:243] Waited 492 ms on lock 0x7b54002891d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:14.605131 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1131438976}
I20260321 02:59:14.634454 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.635140 11380 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:14.635914 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.637120 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:14.639423 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.641001 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.598244 13135 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:14.641554 11307 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:14.643666 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:14.646347 11376 raft_consensus.cc:2393] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 4: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:14.646531 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.647158 11381 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:14.648097 11942 leader_election.cc:304] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:14.609681 13263 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:14.654016 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.657475 11376 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:14.655411 11942 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:14.657835 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.658965 11381 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:14.658290 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.630060 11971 raft_consensus.cc:2468] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:14.657311 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260321 02:59:14.662765 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.665925 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:14.667917 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:14.638839 13276 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:14.606294 13260 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:14.664101 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:59:14.672315 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:14.659241 13263 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:14.674826 13263 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:14.677966 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:14.649245 13135 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:14.680281 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:14.681030 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.682749 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:14.661985 13216 consensus_queue.cc:1059] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260321 02:59:14.684930 13216 raft_consensus.cc:3055] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:59:14.684834 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:14.685462 13216 raft_consensus.cc:740] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:14.686712 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:14.671176 13244 consensus_queue.cc:1048] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260321 02:59:14.691197 11979 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:14.692411 11309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56616 (request call id 1546) took 1685 ms. Trace:
I20260321 02:59:14.679661 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:14.692801 11309 rpcz_store.cc:276] 0321 02:59:13.007199 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:13.587927 (+580728us) service_pool.cc:225] Handling call
0321 02:59:14.692229 (+1104302us) spinlock_profiling.cc:243] Waited 855 ms on lock 0x7b540020c454. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:14.692349 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1965751040}
I20260321 02:59:14.696269 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.696964 11305 raft_consensus.cc:2468] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:14.683773 11379 raft_consensus.cc:2393] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 3: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:14.686475 13216 consensus_queue.cc:260] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:14.699141 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.699810 11304 raft_consensus.cc:2468] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:59:14.702008 11235 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58200 (request call id 1497) took 1521 ms. Trace:
I20260321 02:59:14.702340 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.702437 11235 rpcz_store.cc:276] 0321 02:59:13.181109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:13.181359 (+ 250us) service_pool.cc:225] Handling call
0321 02:59:13.577367 (+396008us) spinlock_profiling.cc:243] Waited 1.27 ms on lock 0x7b5800111398. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b150e760 00007fa8b150e044 00007fa8aaa8f834 00007fa8aaa8f7c2
0321 02:59:14.701672 (+1124305us) spinlock_profiling.cc:243] Waited 970 ms on lock 0x7b540037b254. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:14.701948 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2234210432}
I20260321 02:59:14.703712 11382 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:14.708366 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:14.656832 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:14.708549 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.709997 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:14.660195 13253 raft_consensus.cc:2804] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:59:14.729136 11306 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:14.734679 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.703032 11305 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:14.735288 11311 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:59:14.683723 13235 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:14.730863 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.712394 11276 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:14.739200 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:14.740597 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:14.741868 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:14.742396 11376 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:14.743326 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:14.744153 11381 raft_consensus.cc:2393] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 6: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:14.668321 13194 raft_consensus.cc:2804] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:14.747749 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:14.748466 11227 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:14.689939 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:14.750209 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:14.752370 11235 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:14.759061 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.759768 11382 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
W20260321 02:59:14.762501 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 59856.5us
W20260321 02:59:14.770826 11349 connection.cc:232] RPC call timeout handler was delayed by -1.1584262919999446s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19540000000000002s for a response
W20260321 02:59:14.771200 11349 connection.cc:232] RPC call timeout handler was delayed by -1.1521262919999344s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19610000000000002s for a response
W20260321 02:59:14.771430 11349 connection.cc:232] RPC call timeout handler was delayed by -1.1026262919999681s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2016s for a response
I20260321 02:59:14.745630 11974 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:14.773231 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.770350 11305 raft_consensus.cc:2955] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } } }
I20260321 02:59:14.734396 13226 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:14.774061 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.735129 13253 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:14.777143 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.778765 11381 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:14.766400 11275 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:14.764107 11973 raft_consensus.cc:2393] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 4: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
I20260321 02:59:14.788061 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.788951 11380 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
W20260321 02:59:14.771692 11349 connection.cc:232] RPC call timeout handler was delayed by -1.1026262919999681s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2016s for a response
I20260321 02:59:14.782090 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.782835 11379 raft_consensus.cc:1240] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:14.785343 11275 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:14.790697 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.791383 11275 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:14.781057 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.785535 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:14.789752 11349 connection.cc:232] RPC call timeout handler was delayed by -1.0562109739998959s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1646s for a response
W20260321 02:59:14.796037 11349 connection.cc:232] RPC call timeout handler was delayed by -1.0562109739998959s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1646s for a response
I20260321 02:59:14.791498 11376 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:14.797708 11227 raft_consensus.cc:3060] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:14.797433 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.799371 11382 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:14.788054 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.796739 11275 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:14.759485 13235 raft_consensus.cc:2804] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:14.802804 11275 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:14.791256 11308 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
W20260321 02:59:14.796772 11349 connection.cc:232] RPC call timeout handler was delayed by -1.0510454249998702s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16690000000000002s for a response
I20260321 02:59:14.801327 11309 raft_consensus.cc:2393] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 6: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
W20260321 02:59:14.804112 11349 connection.cc:232] RPC call timeout handler was delayed by -1.0441262919999872s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2081s for a response
I20260321 02:59:14.805126 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.805750 11302 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:14.804086 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.808446 11349 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:14.809580 11349 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:14.765201 13194 raft_consensus.cc:697] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:14.813388 13086 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:14.811851 11349 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:14.811120 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.816591 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.787264 13253 raft_consensus.cc:3060] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:14.751477 13084 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:14.820225 11349 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:14.822052 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.813565 13178 raft_consensus.cc:2804] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:14.862309 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:14.863226 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:14.804939 13272 raft_consensus.cc:2804] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:14.863415 11228 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:14.866346 11378 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:14.823915 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.818825 11302 raft_consensus.cc:2410] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260321 02:59:14.837692 11235 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:14.863282 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:14.868618 11236 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:14.860775 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.871830 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:14.872589 13260 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:14.872792 13203 raft_consensus.cc:3060] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:14.875315 11227 raft_consensus.cc:2468] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:14.877208 13188 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:14.867532 13272 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:14.878346 13272 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:14.879284 11228 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:14.826421 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:14.833704 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.793211 13210 raft_consensus.cc:2804] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:14.809580 13139 raft_consensus.cc:2749] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:14.805240 13242 raft_consensus.cc:2749] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260321 02:59:14.865967 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 45864.8us
W20260321 02:59:14.886945 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:14.880678 11233 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:14.807484 13235 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:14.817530 13093 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:14.891924 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.885511 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.784736 13220 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:14.900043 11308 raft_consensus.cc:2393] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 6: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:14.889708 11234 raft_consensus.cc:1240] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:14.901281 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.815490 13226 raft_consensus.cc:2749] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:14.890763 13235 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:14.825184 13216 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:14.862892 13178 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:14.825994 13277 raft_consensus.cc:2749] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:14.880189 13262 raft_consensus.cc:2804] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:14.887607 11349 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:14.910691 11302 raft_consensus.cc:1240] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:14.883898 13210 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:14.911121 11349 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:14.914350 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.921901 11978 raft_consensus.cc:2393] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 5: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
I20260321 02:59:14.922446 11349 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:14.919634 11232 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:14.925382 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.926271 11376 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:14.934140 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.939245 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:14.906409 13093 raft_consensus.cc:697] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:14.912968 12961 raft_consensus.cc:2804] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:14.915730 13178 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:14.913916 13262 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:14.913216 13210 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:14.949565 11381 raft_consensus.cc:1240] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:14.927011 13138 raft_consensus.cc:2804] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:14.966027 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.813403 13194 consensus_queue.cc:237] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:14.954298 11977 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45098 (request call id 307) took 1246 ms. Trace:
I20260321 02:59:14.941627 13277 raft_consensus.cc:2804] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:14.982793 13093 consensus_queue.cc:237] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:14.983834 13294 log.cc:927] Time spent T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.224s user 0.001s sys 0.000s
I20260321 02:59:14.956079 11376 raft_consensus.cc:2955] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } } }
W20260321 02:59:14.985635 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 330448us
I20260321 02:59:14.949216 12961 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:14.992579 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:14.957854 13262 raft_consensus.cc:3060] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:14.971231 13295 log.cc:927] Time spent T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.261s user 0.001s sys 0.000s
I20260321 02:59:14.992316 11200 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:14.995328 11942 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:14.980171 13138 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:14.982649 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:14.987627 13277 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:15.008347 13138 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:15.014573 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:14.995466 12961 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
W20260321 02:59:15.023372 13297 log.cc:927] Time spent T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20260321 02:59:15.026072 13123 raft_consensus.cc:3055] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:15.026544 13123 raft_consensus.cc:740] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:15.003567 13208 raft_consensus.cc:2749] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:15.027381 13123 consensus_queue.cc:260] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:15.031147 13123 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:14.985127 11977 rpcz_store.cc:276] 0321 02:59:13.707656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:13.711029 (+ 3373us) service_pool.cc:225] Handling call
0321 02:59:14.954043 (+1243014us) spinlock_profiling.cc:243] Waited 708 ms on lock 0x7b540046dcd4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:14.954226 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1627407232}
I20260321 02:59:15.004658 13011 raft_consensus.cc:2804] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:15.035157 13011 raft_consensus.cc:697] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:15.035647 13253 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:15.036196 13011 consensus_queue.cc:237] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:15.020031 13277 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:15.006067 12720 consensus_queue.cc:1059] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.061s
I20260321 02:59:15.090703 13220 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:15.067160 13242 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:15.093628 13272 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.098781 13139 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:15.100812 13262 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.104838 13253 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:15.105935 13235 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:15.113225 13135 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:15.118147 11236 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
W20260321 02:59:15.122051 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 319283us
W20260321 02:59:15.107832 13264 consensus_meta.cc:287] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.536s user 0.006s sys 0.002s
I20260321 02:59:15.128489 11275 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:15.136292 11310 raft_consensus.cc:1240] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:15.137137 11275 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:15.126659 13203 raft_consensus.cc:3055] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:15.139308 11275 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:15.141451 11350 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:15.134552 13264 raft_consensus.cc:2749] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285) with higher term. Message: Invalid argument: T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4.
I20260321 02:59:15.126947 13139 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.084831 13084 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:15.140450 13263 raft_consensus.cc:2804] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:15.124271 13242 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.142658 13123 raft_consensus.cc:2749] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:15.141395 13203 raft_consensus.cc:740] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:15.152253 11377 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e"
dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:40934
W20260321 02:59:15.170265 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 168895us
W20260321 02:59:15.130364 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 205204us
I20260321 02:59:15.177379 11303 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:15.159830 13263 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:15.193064 11383 raft_consensus.cc:1240] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
W20260321 02:59:15.194412 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 55150.9us
I20260321 02:59:15.194962 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:15.195271 11275 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:15.198330 11275 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:15.199280 11200 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:15.208055 11275 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:15.211478 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 208845us
W20260321 02:59:15.220968 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 47430.6us
I20260321 02:59:15.221649 13306 consensus_queue.cc:1048] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:59:15.222950 11349 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:15.224118 11349 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:15.197069 13263 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:15.218071 11275 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:15.228953 11200 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:15.230995 11200 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:15.225359 11942 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:15.205538 13301 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:15.208176 13260 raft_consensus.cc:2749] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:15.238432 11275 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:15.212659 12720 raft_consensus.cc:2749] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:15.245069 11383 raft_consensus.cc:1275] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260321 02:59:15.218556 13123 raft_consensus.cc:2749] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:15.242791 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:15.213755 13093 raft_consensus.cc:2804] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:15.209385 13096 consensus_queue.cc:1059] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260321 02:59:15.185859 13178 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.207093 13176 consensus_queue.cc:1059] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260321 02:59:15.247210 13253 raft_consensus.cc:2804] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:15.238142 13208 raft_consensus.cc:2749] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:15.250578 13276 raft_consensus.cc:2804] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260321 02:59:15.279983 13300 log.cc:927] Time spent T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.141s user 0.000s sys 0.001s
W20260321 02:59:15.281371 13311 log.cc:927] Time spent T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.118s user 0.002s sys 0.000s
I20260321 02:59:15.272993 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
W20260321 02:59:15.282351 13309 log.cc:927] Time spent T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260321 02:59:15.278855 13210 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.255270 13260 raft_consensus.cc:2804] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:15.284852 11972 raft_consensus.cc:1240] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:15.285149 13260 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:15.253255 13093 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:15.255384 13135 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:15.285912 13093 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:15.286371 13178 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:15.287257 13210 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:15.284312 13276 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:15.284305 13304 log.cc:927] Time spent T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.135s user 0.000s sys 0.001s
I20260321 02:59:15.285563 13260 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:15.227757 12961 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.254611 13262 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:15.298894 11377 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:59:15.304593 11377 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:15.216455 13203 consensus_queue.cc:260] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:15.300362 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 62119.5us
I20260321 02:59:15.302549 12720 raft_consensus.cc:2804] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:15.310137 12720 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:15.311116 12720 raft_consensus.cc:3060] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:15.312583 13291 raft_consensus.cc:3055] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:15.313129 13291 raft_consensus.cc:740] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
W20260321 02:59:15.310240 13313 log.cc:927] Time spent T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.102s user 0.000s sys 0.001s
I20260321 02:59:15.269308 13253 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
W20260321 02:59:15.326424 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 95534.9us
I20260321 02:59:15.328248 11200 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:15.318440 11275 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:15.330112 11349 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:15.332295 11275 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:15.332638 13139 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:15.301551 13276 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:15.337585 11275 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:15.271245 13220 consensus_queue.cc:1048] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
W20260321 02:59:15.348201 11350 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 206835us
I20260321 02:59:15.306365 13301 raft_consensus.cc:2749] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:15.348291 11349 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:15.347616 11200 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:15.314456 13291 consensus_queue.cc:260] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.333745 13253 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:15.361091 11275 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:15.336807 13178 raft_consensus.cc:2804] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:15.341648 13210 raft_consensus.cc:2804] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:59:15.350023 13076 raft_consensus.cc:2804] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:15.346711 13203 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
W20260321 02:59:15.378242 13316 log.cc:927] Time spent T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.114s user 0.000s sys 0.001s
I20260321 02:59:15.379534 13301 raft_consensus.cc:2804] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:15.380149 13301 raft_consensus.cc:697] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:15.385201 13210 raft_consensus.cc:697] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:15.386337 13210 consensus_queue.cc:237] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.388154 11309 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:59:15.369082 11377 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.370636 11275 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:15.346915 13123 raft_consensus.cc:2749] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260321 02:59:15.358690 13305 log.cc:927] Time spent T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20260321 02:59:15.375483 13076 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:15.397979 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.398424 11275 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:15.398535 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.355938 13319 raft_consensus.cc:2804] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:15.363317 13314 raft_consensus.cc:2749] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260321 02:59:15.404192 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:15.416412 11978 raft_consensus.cc:1240] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 1. Current term is 2. Ops: []
I20260321 02:59:15.403898 11275 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:15.357067 13176 consensus_queue.cc:1059] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.045s
W20260321 02:59:15.420109 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 71901.5us
I20260321 02:59:15.421093 11349 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:15.408890 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.410035 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.410295 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.415266 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.365841 13178 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:15.389681 13139 raft_consensus.cc:2749] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:15.418828 11275 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:15.407740 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.408813 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.420020 11236 raft_consensus.cc:1240] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:15.423338 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:15.423441 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:15.424139 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:15.413339 13319 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:15.424623 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:15.425159 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.425449 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.424656 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.399550 13291 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
W20260321 02:59:15.476472 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 55437.3us
I20260321 02:59:15.477497 11349 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:15.381098 13301 consensus_queue.cc:237] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.413262 13076 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:15.416029 13135 raft_consensus.cc:2804] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:15.404006 13216 raft_consensus.cc:2804] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:15.403895 13306 consensus_queue.cc:1059] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.069s
I20260321 02:59:15.482908 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:15.483588 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:15.482925 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.420686 13328 raft_consensus.cc:2749] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:15.428324 13123 raft_consensus.cc:2804] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:15.471597 13176 raft_consensus.cc:3055] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:15.515841 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.431382 13178 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:15.419867 13315 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:15.430305 12961 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:15.478905 13319 raft_consensus.cc:3060] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:15.486613 13216 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:15.482071 13135 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:15.520823 11230 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:15.555271 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.479485 13335 raft_consensus.cc:2764] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20260321 02:59:15.528188 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 109426us
I20260321 02:59:15.549358 12720 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.498375 13123 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:15.523537 13139 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:15.564024 11275 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:15.566150 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.557883 13135 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:15.566334 11385 raft_consensus.cc:1275] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260321 02:59:15.561129 13323 log.cc:927] Time spent T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.237s user 0.002s sys 0.000s
I20260321 02:59:15.553833 13216 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:15.520303 13176 raft_consensus.cc:740] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:15.604700 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:15.521349 13339 raft_consensus.cc:2749] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:15.573602 11233 raft_consensus.cc:2393] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 4: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:15.482987 13333 consensus_queue.cc:1059] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.075s
I20260321 02:59:15.572430 13123 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:15.611603 13314 raft_consensus.cc:3055] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Stepping down as leader of term 1
I20260321 02:59:15.612310 13314 raft_consensus.cc:740] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:15.613216 13314 consensus_queue.cc:260] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:15.614660 13314 raft_consensus.cc:3060] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:15.616562 13086 raft_consensus.cc:2804] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:15.617044 13086 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:15.617476 13086 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:15.565773 13242 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:15.621541 13096 raft_consensus.cc:3055] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:59:15.622023 13096 raft_consensus.cc:740] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:15.622432 13176 consensus_queue.cc:260] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.622896 13096 consensus_queue.cc:260] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.624275 13176 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:15.624508 11234 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58214 (request call id 1124) took 1515 ms. Trace:
I20260321 02:59:15.624966 11234 rpcz_store.cc:276] 0321 02:59:14.109165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:14.211412 (+102247us) service_pool.cc:225] Handling call
0321 02:59:14.227107 (+ 15695us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:14.901605 (+674498us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:15.624241 (+722636us) spinlock_profiling.cc:243] Waited 662 ms on lock 0x7b54003b8b54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:15.624443 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1523367168}
I20260321 02:59:15.570856 13264 raft_consensus.cc:2804] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260321 02:59:15.628870 13330 log.cc:927] Time spent T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260321 02:59:15.629099 13272 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:15.616554 13194 consensus_queue.cc:1048] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260321 02:59:15.573051 13139 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:15.561111 13328 raft_consensus.cc:3055] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:15.558085 13310 raft_consensus.cc:2804] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:15.562947 13315 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:15.647488 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 129696us
I20260321 02:59:15.656857 11349 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:15.600409 11275 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:15.659019 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:15.659631 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 59382.5us
I20260321 02:59:15.662498 13342 raft_consensus.cc:2804] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:15.662950 13342 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:15.663304 13342 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:15.665908 11275 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:15.667032 13272 raft_consensus.cc:2804] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:15.667282 13264 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:15.667564 13272 raft_consensus.cc:697] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:15.649529 13093 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:15.668798 11303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49786 (request call id 147) took 1033 ms. Trace:
I20260321 02:59:15.698063 11303 rpcz_store.cc:276] 0321 02:59:14.635451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:14.636888 (+ 1437us) service_pool.cc:225] Handling call
0321 02:59:15.668563 (+1031675us) spinlock_profiling.cc:243] Waited 513 ms on lock 0x7b5400319554. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:15.668714 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1180776320}
I20260321 02:59:15.695488 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:15.700775 11234 raft_consensus.cc:2393] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 4: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:15.701480 11233 raft_consensus.cc:2393] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 4: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:15.703729 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:15.704638 11235 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:15.668519 13272 consensus_queue.cc:237] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.706491 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:15.707787 11310 raft_consensus.cc:1275] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260321 02:59:15.708182 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:15.689348 13138 consensus_meta.cc:287] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.623s user 0.011s sys 0.001s
I20260321 02:59:15.708979 13138 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.709302 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:15.709884 11306 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:15.710305 13226 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:15.707832 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.696452 11377 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:15.716861 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:15.717731 11303 raft_consensus.cc:2376] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4. Re-sending same reply.
W20260321 02:59:15.721467 13338 log.cc:927] Time spent T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
W20260321 02:59:15.722388 13277 consensus_meta.cc:287] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.598s user 0.011s sys 0.001s
I20260321 02:59:15.722787 13277 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.725739 13076 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.727233 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:15.727711 11380 raft_consensus.cc:1240] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:15.699472 11231 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:15.650120 13139 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:15.736835 11275 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:15.768095 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:15.728314 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:15.768990 11303 raft_consensus.cc:2393] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:15.769145 11381 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 5
I20260321 02:59:15.769439 11306 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:15.728749 11383 raft_consensus.cc:1275] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:15.738693 11229 raft_consensus.cc:1275] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:15.741916 11349 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:15.650363 13328 raft_consensus.cc:740] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:15.771281 11349 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:15.729938 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:15.730216 11230 raft_consensus.cc:2368] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4.
I20260321 02:59:15.733186 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:15.784821 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:15.754961 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:15.766448 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.667923 13264 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:15.728820 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:15.686336 13308 log.cc:927] Time spent T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.525s user 0.003s sys 0.000s
I20260321 02:59:15.655361 13310 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:15.772336 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:15.729452 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:15.828135 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.828855 11227 raft_consensus.cc:2468] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 6.
I20260321 02:59:15.773768 13306 raft_consensus.cc:2804] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:15.773862 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:15.783416 11235 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:15.786278 11233 raft_consensus.cc:2393] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 4: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:15.768006 11275 leader_election.cc:304] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:15.787039 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:15.836288 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 68413.7us
I20260321 02:59:15.775794 13235 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:15.799330 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:15.800238 11234 raft_consensus.cc:2393] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 3: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
W20260321 02:59:15.849598 11350 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 496738us
I20260321 02:59:15.807404 11381 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:15.816915 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.852836 11227 raft_consensus.cc:1275] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260321 02:59:15.710991 13138 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:15.712616 13325 raft_consensus.cc:2749] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:15.695636 13096 raft_consensus.cc:3060] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:15.694947 13093 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:15.871083 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:15.773934 13342 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.727932 13076 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:15.710937 13226 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.877882 11380 raft_consensus.cc:1240] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 2. Current term is 3. Ops: []
I20260321 02:59:15.840942 13306 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:15.788103 13315 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:15.842236 11275 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:15.885779 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 43707.7us
I20260321 02:59:15.886540 11275 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:15.789809 11349 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:15.842998 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:15.843667 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:15.887785 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 98105us
I20260321 02:59:15.818182 13335 raft_consensus.cc:2804] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:15.853456 11232 raft_consensus.cc:2468] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:15.829617 13310 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:15.858189 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.901190 11942 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:15.816303 13341 raft_consensus.cc:2749] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:15.864362 11381 raft_consensus.cc:1275] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260321 02:59:15.875895 13235 raft_consensus.cc:2749] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260321 02:59:15.868072 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:15.907670 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:15.869212 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.872256 11377 raft_consensus.cc:2393] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 4: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:15.707788 13242 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:15.834439 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:15.889235 11233 raft_consensus.cc:2393] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:15.891515 11349 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:15.879179 13138 consensus_queue.cc:1059] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260321 02:59:15.905234 11235 raft_consensus.cc:2468] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 2.
I20260321 02:59:15.917685 11384 raft_consensus.cc:1240] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:15.921378 11227 raft_consensus.cc:2393] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 2: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:15.900041 13335 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:15.921926 13335 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:15.924108 13194 consensus_queue.cc:1048] T 006e9dd81c0c4368bed549c97128f4ea P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260321 02:59:15.927060 13135 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:15.975035 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:15.918991 13307 consensus_queue.cc:1048] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.078s
I20260321 02:59:15.897859 13325 raft_consensus.cc:2804] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:15.930356 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:15.931075 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:15.983722 11381 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:15.937031 11377 raft_consensus.cc:1275] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20260321 02:59:15.951887 13276 consensus_meta.cc:287] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.595s user 0.009s sys 0.000s
I20260321 02:59:15.937675 13242 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:15.908911 11229 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:15.939144 13138 raft_consensus.cc:3055] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:15.874444 13328 consensus_queue.cc:260] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:15.941601 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 50170.3us
I20260321 02:59:15.942247 13093 raft_consensus.cc:2749] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:15.921845 11380 raft_consensus.cc:1240] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:15.984819 13325 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:15.886415 12720 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:15.887158 13306 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:15.978252 11200 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:16.063868 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 85739.2us
I20260321 02:59:15.949237 13086 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.907349 13220 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:16.064586 11200 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:15.929370 13319 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.954084 13342 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:15.979604 13134 raft_consensus.cc:993] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb: : Instructing follower 83b12b883a7f4129a87dadf5a78f08da to start an election
I20260321 02:59:15.954289 13277 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:15.980675 13341 raft_consensus.cc:2749] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:15.966445 13315 consensus_queue.cc:1048] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260321 02:59:15.989957 13276 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.001000 11134 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42530 (request call id 218) took 6264 ms. Trace:
W20260321 02:59:16.069205 13260 consensus_meta.cc:287] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.744s user 0.012s sys 0.002s
I20260321 02:59:15.982671 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:16.002154 11130 catalog_manager.cc:5671] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 2 to 3, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4). New cstate: current_term: 3 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260321 02:59:16.003206 13263 consensus_meta.cc:287] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.760s user 0.010s sys 0.002s
I20260321 02:59:16.003293 13226 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:15.956701 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:15.984035 13235 raft_consensus.cc:2804] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:16.001048 13216 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:15.989043 13210 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:15.981315 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:16.066300 11381 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:15.990022 13138 raft_consensus.cc:740] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:16.079782 11200 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:16.083248 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 196771us
I20260321 02:59:16.062557 13325 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:16.001111 13135 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:16.085834 11134 rpcz_store.cc:276] 0321 02:59:09.736217 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:09.736512 (+ 295us) service_pool.cc:225] Handling call
0321 02:59:16.000937 (+6264425us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:59:14.745540 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:59:14.903057 (+157517us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:59:14.904857 (+ 1800us) write_op.cc:435] Acquired schema lock
0321 02:59:14.906710 (+ 1853us) tablet.cc:661] Decoding operations
0321 02:59:14.985627 (+ 78917us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:59:14.988319 (+ 2692us) write_op.cc:641] Partition lock acquired for write op
0321 02:59:14.990309 (+ 1990us) tablet.cc:684] Acquiring locks for 2 operations
0321 02:59:15.052529 (+ 62220us) tablet.cc:700] Row locks acquired
0321 02:59:15.068924 (+ 16395us) write_op.cc:260] PREPARE: finished
0321 02:59:15.088223 (+ 19299us) write_op.cc:270] Start()
0321 02:59:15.088401 (+ 178us) write_op.cc:276] Timestamp: P: 1774061955084874 usec, L: 0
0321 02:59:15.088435 (+ 34us) op_driver.cc:348] REPLICATION: starting
0321 02:59:15.089472 (+ 1037us) log.cc:844] Serialized 913 byte log entry
0321 02:59:15.400286 (+310814us) op_driver.cc:464] REPLICATION: finished
0321 02:59:15.593197 (+192911us) write_op.cc:301] APPLY: starting
0321 02:59:15.599787 (+ 6590us) tablet.cc:1366] starting BulkCheckPresence
0321 02:59:15.647568 (+ 47781us) tablet.cc:1369] finished BulkCheckPresence
0321 02:59:15.661073 (+ 13505us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:59:15.701689 (+ 40616us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:59:15.701910 (+ 221us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0321 02:59:15.701980 (+ 70us) write_op.cc:312] APPLY: finished
0321 02:59:15.702234 (+ 254us) log.cc:844] Serialized 29 byte log entry
0321 02:59:15.702439 (+ 205us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:15.702538 (+ 99us) write_op.cc:454] Released schema lock
0321 02:59:15.702579 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0321 02:59:15.702932 (+ 353us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:15.702970 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":580,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":178944,"apply.run_cpu_time_us":3731,"apply.run_wall_time_us":109738,"num_ops":2,"prepare.queue_time_us":442526,"prepare.run_cpu_time_us":19035,"prepare.run_wall_time_us":468340,"raft.queue_time_us":57659,"raft.run_cpu_time_us":29947,"raft.run_wall_time_us":207865,"replication_time_us":280168,"spinlock_wait_cycles":2304,"thread_start_us":529248,"threads_started":4,"wal-append.queue_time_us":192933}]]}
I20260321 02:59:16.078233 13291 raft_consensus.cc:2804] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:16.119015 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:16.120600 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:16.046363 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 52766.8us
I20260321 02:59:16.083930 13244 raft_consensus.cc:1081] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Signalling peer 83b12b883a7f4129a87dadf5a78f08da to start an election
I20260321 02:59:16.083709 13176 raft_consensus.cc:2764] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260321 02:59:16.090526 11130 catalog_manager.cc:5671] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:59:16.106828 11200 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:16.126703 11306 raft_consensus.cc:1240] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:16.108296 11379 raft_consensus.cc:1240] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:16.131455 13276 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:16.119567 13291 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:16.134016 13291 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:16.135933 13138 consensus_queue.cc:260] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:16.138083 13138 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:16.138661 13352 log.cc:927] Time spent T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260321 02:59:16.140283 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:16.141144 11303 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:16.093705 13263 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.143436 13263 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:16.112298 13210 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.144733 13242 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:16.145829 13210 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:16.119601 13253 consensus_meta.cc:287] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 0.746s user 0.013s sys 0.000s
I20260321 02:59:16.148679 13253 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:16.150992 13253 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:16.153180 13347 log.cc:927] Time spent T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20260321 02:59:16.154570 13216 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:16.095470 13235 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:16.155562 13235 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:16.157060 13319 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:16.111114 13086 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:16.158596 13123 consensus_meta.cc:287] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.547s user 0.010s sys 0.001s
I20260321 02:59:16.159051 13123 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.113224 11308 raft_consensus.cc:1240] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:16.119084 12720 raft_consensus.cc:2749] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:16.161546 11975 raft_consensus.cc:1240] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:16.163000 13335 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.114293 11275 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:16.114104 11305 raft_consensus.cc:1240] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:16.166222 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:16.167387 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:16.164639 11200 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:16.170198 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 55962.5us
I20260321 02:59:16.164287 13339 consensus_queue.cc:1059] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20260321 02:59:16.171604 11275 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:16.172432 13339 raft_consensus.cc:3055] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:16.172585 11275 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:16.172834 13339 raft_consensus.cc:740] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:16.173669 13086 raft_consensus.cc:2749] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:16.173650 13339 consensus_queue.cc:260] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:16.174813 13086 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:16.174891 13331 raft_consensus.cc:2804] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:16.173858 11275 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:16.182083 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:16.185405 11200 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:16.092449 13260 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:16.191304 11200 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:16.196344 11349 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:16.179349 13220 raft_consensus.cc:2749] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:16.175815 13086 raft_consensus.cc:697] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:16.200173 13086 consensus_queue.cc:237] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.202463 13325 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.204505 13123 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:16.166985 13084 raft_consensus.cc:2749] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260321 02:59:16.207151 13350 log.cc:927] Time spent T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.088s user 0.001s sys 0.001s
I20260321 02:59:16.209448 13176 raft_consensus.cc:2749] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:16.212954 13260 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:16.175769 13331 raft_consensus.cc:697] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:16.219280 11349 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:16.195420 11200 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:16.233566 11200 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:16.235853 11349 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:16.241201 11305 raft_consensus.cc:1240] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:16.250684 11200 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:16.176882 11972 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:16.175105 13339 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:16.194715 13244 raft_consensus.cc:2749] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:16.256727 11200 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:16.259792 11200 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:16.216580 12720 raft_consensus.cc:2804] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:16.273298 11308 raft_consensus.cc:1240] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:16.274591 11350 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:16.248816 13328 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:16.249341 13307 raft_consensus.cc:2749] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260321 02:59:16.285066 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 47457us
I20260321 02:59:16.253468 13176 raft_consensus.cc:2749] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:16.280866 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:16.178318 13264 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:16.273234 13178 consensus_meta.cc:287] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.710s user 0.002s sys 0.009s
I20260321 02:59:16.284587 11977 raft_consensus.cc:2468] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
W20260321 02:59:16.323164 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 149407us
I20260321 02:59:16.313791 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:16.325736 11275 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:16.308863 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:16.314260 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:16.327106 11311 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:16.257774 13192 raft_consensus.cc:2804] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:16.335590 11303 raft_consensus.cc:3060] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:16.252696 13084 raft_consensus.cc:2749] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:16.272771 13096 raft_consensus.cc:2749] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:16.344519 11275 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:16.324633 13139 consensus_meta.cc:287] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.536s user 0.010s sys 0.001s
W20260321 02:59:16.352097 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 64423.3us
I20260321 02:59:16.354157 11349 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:16.358935 11303 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:16.364308 11349 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:16.282474 13253 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:16.274154 12720 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:16.335139 13192 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:16.288365 13319 consensus_queue.cc:1059] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.064s
I20260321 02:59:16.368216 11275 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:16.328123 13331 consensus_queue.cc:237] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:16.285794 13327 raft_consensus.cc:2804] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:16.288542 13244 raft_consensus.cc:2749] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:16.342595 13357 raft_consensus.cc:2804] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:16.365587 11972 raft_consensus.cc:1240] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:16.365775 13176 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:16.328104 13178 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.355312 11975 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45088 (request call id 253) took 1119 ms. Trace:
I20260321 02:59:16.365186 13307 raft_consensus.cc:2764] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260321 02:59:16.404599 11311 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:16.380553 13123 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:16.381752 13335 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:16.412178 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 44065.8us
I20260321 02:59:16.414143 13325 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:16.381966 11276 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:16.380751 13260 consensus_queue.cc:1048] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260321 02:59:16.409107 12720 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:16.406157 13272 raft_consensus.cc:2764] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260321 02:59:16.396387 12961 raft_consensus.cc:2749] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:16.377493 13306 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.441294 13306 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:16.385926 13361 consensus_queue.cc:1059] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260321 02:59:16.443049 13361 raft_consensus.cc:3055] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:16.443467 13361 raft_consensus.cc:740] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:16.444321 13361 consensus_queue.cc:260] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:16.445720 13138 raft_consensus.cc:3055] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:16.446123 13361 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:16.450711 13123 raft_consensus.cc:697] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:16.446195 13138 raft_consensus.cc:740] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:16.420720 13357 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:16.425571 13327 raft_consensus.cc:697] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:16.453305 13138 consensus_queue.cc:260] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.431102 11975 rpcz_store.cc:276] 0321 02:59:15.234390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:15.236925 (+ 2535us) service_pool.cc:225] Handling call
0321 02:59:15.665119 (+428194us) spinlock_profiling.cc:243] Waited 6.23 ms on lock 0x7b58002c3798. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b1522680 00007fa8b150cfe9 00007fa8b150ca92 00007fa8aaa8ad34
0321 02:59:15.665397 (+ 278us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:16.161957 (+496560us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:16.336105 (+174148us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b5400701054. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:16.353499 (+ 17394us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1154566144}
I20260321 02:59:16.454528 13327 consensus_queue.cc:237] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:16.365927 13139 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.408576 13277 raft_consensus.cc:2749] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:16.410884 13192 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:16.458462 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 76608.3us
I20260321 02:59:16.411130 13253 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:16.460062 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 178204us
I20260321 02:59:16.458220 13138 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:16.425782 13086 raft_consensus.cc:2804] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:16.453352 13357 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:16.462957 13260 raft_consensus.cc:2804] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election won for term 3
I20260321 02:59:16.477166 13086 raft_consensus.cc:697] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:16.478081 13139 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:16.460898 13178 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:16.430882 13176 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.465209 13272 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:16.483896 12720 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:16.499575 13086 consensus_queue.cc:237] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.504856 13203 consensus_queue.cc:1048] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:16.505568 13337 consensus_queue.cc:1059] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:59:16.484027 13260 raft_consensus.cc:697] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:16.507649 11200 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:16.518882 11182 auto_rebalancer.cc:274] failed to send replica move request: Illegal state: Replica 0d7efc590a1041a9bd4cd4a971332dfb is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:16.490993 13264 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:16.484148 13253 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:16.523203 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 223205us
I20260321 02:59:16.524920 13076 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:16.458103 13310 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:16.507309 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 75167.2us
I20260321 02:59:16.548458 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:16.548987 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:16.521833 13272 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:16.549474 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:16.552971 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:16.553401 11303 raft_consensus.cc:2393] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 6: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:16.553900 11306 raft_consensus.cc:2393] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 6: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:16.549988 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:16.555174 11308 raft_consensus.cc:2376] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6. Re-sending same reply.
I20260321 02:59:16.556735 11307 raft_consensus.cc:2393] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:16.476204 13123 consensus_queue.cc:237] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.562417 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:16.544884 13375 raft_consensus.cc:2804] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:16.565413 11305 raft_consensus.cc:1275] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260321 02:59:16.566339 11378 raft_consensus.cc:1240] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:16.567087 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:16.564407 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:16.507562 13235 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.589536 11310 raft_consensus.cc:2393] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 6: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:16.556041 13076 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.596520 12720 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:16.597952 13076 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:16.543530 13084 consensus_queue.cc:1059] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.037s
I20260321 02:59:16.598518 11308 raft_consensus.cc:1240] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:16.599068 13084 raft_consensus.cc:3055] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:16.599503 13084 raft_consensus.cc:740] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:16.600199 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:16.600339 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:16.600878 11303 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:16.601263 11304 raft_consensus.cc:1240] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:16.602277 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:16.602850 11304 raft_consensus.cc:3060] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:16.601303 11307 raft_consensus.cc:2393] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:16.604211 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:16.604388 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:16.604393 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:16.605043 11307 raft_consensus.cc:2393] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:16.605376 11306 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:16.606119 11380 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:16.607049 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:16.612310 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:16.608065 11307 raft_consensus.cc:2393] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 4: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:16.613168 11309 raft_consensus.cc:2393] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 4: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:16.605178 11311 raft_consensus.cc:2393] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:16.600301 13084 consensus_queue.cc:260] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.555405 13328 consensus_queue.cc:1059] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260321 02:59:16.617980 11308 raft_consensus.cc:1240] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260321 02:59:16.625818 11304 raft_consensus.cc:2468] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:16.619639 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:16.625977 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 84597.6us
I20260321 02:59:16.628542 11200 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:16.628616 11942 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:16.630156 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:16.630784 11230 raft_consensus.cc:2468] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:16.631711 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:16.619982 11306 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:16.622639 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:16.637677 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:16.638823 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:16.639106 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 58482.8us
I20260321 02:59:16.623577 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:16.599103 13310 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:16.644501 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:16.645303 11311 raft_consensus.cc:2393] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 6: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:16.645161 11200 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:16.646637 11303 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:16.641139 11227 raft_consensus.cc:1275] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260321 02:59:16.591656 13375 raft_consensus.cc:697] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:16.644248 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:16.625119 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:16.554312 13192 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.648628 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:16.658337 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:16.656677 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:16.648850 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:16.641426 11349 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:16.658699 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:16.662179 11349 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:16.663810 11275 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:16.664270 11307 raft_consensus.cc:2393] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:16.659559 11310 raft_consensus.cc:2393] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:16.666911 11275 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:16.670718 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:16.670720 11232 raft_consensus.cc:1275] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:59:16.676520 11275 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:16.648083 13076 raft_consensus.cc:2749] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:16.678156 13384 consensus_queue.cc:1059] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:59:16.678274 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:16.678589 13235 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:16.652266 13084 raft_consensus.cc:3060] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:16.652952 13203 raft_consensus.cc:2804] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:16.679580 11230 raft_consensus.cc:1240] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:16.670930 11349 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:16.648845 13194 consensus_queue.cc:1048] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260321 02:59:16.689131 11275 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:16.670094 13210 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:16.702239 11182 auto_rebalancer.cc:791] Removing replace marker failed: Replica 0d7efc590a1041a9bd4cd4a971332dfb is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:16.641283 13291 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.684417 13176 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:16.701969 11349 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:16.669569 13310 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:16.668354 13253 raft_consensus.cc:2804] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader election won for term 4
W20260321 02:59:16.703871 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 75385.4us
I20260321 02:59:16.663298 12720 consensus_queue.cc:1059] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20260321 02:59:16.632025 13260 consensus_queue.cc:237] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:16.682029 13216 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:16.740329 11349 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:16.706125 11182 auto_rebalancer.cc:799] Could not move replica: Incomplete: tablet 8a38bdd851a24a9a9c5e835ac457cda2, TS 83b12b883a7f4129a87dadf5a78f08da -> TS 3d8d3176d6ec407ea2a965556c2e2c56 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.743044 11231 raft_consensus.cc:1240] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:16.673265 13178 raft_consensus.cc:2749] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260321 02:59:16.743170 11182 auto_rebalancer.cc:284] scheduled replica move failed to complete: Incomplete: tablet 8a38bdd851a24a9a9c5e835ac457cda2, TS 83b12b883a7f4129a87dadf5a78f08da -> TS 3d8d3176d6ec407ea2a965556c2e2c56 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:16.740195 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 51138us
I20260321 02:59:16.744884 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:16.746439 11275 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:16.688380 13353 raft_consensus.cc:2749] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260321 02:59:16.725425 13377 log.cc:927] Time spent T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.190s user 0.000s sys 0.001s
I20260321 02:59:16.701752 13203 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:16.679864 13337 consensus_queue.cc:1059] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.059s
I20260321 02:59:16.750417 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:16.707554 13328 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election won for term 6
W20260321 02:59:16.759301 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:16.751268 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:16.768774 11381 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:16.769026 11942 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:16.759804 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:16.762359 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:16.763355 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:16.748807 11275 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:16.765823 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:16.764334 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:16.739543 13307 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:16.766696 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:16.745715 13301 raft_consensus.cc:2749] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:16.773731 11384 raft_consensus.cc:1275] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260321 02:59:16.741365 13325 consensus_queue.cc:1048] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.046s
I20260321 02:59:16.685390 13355 consensus_queue.cc:1048] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260321 02:59:16.807205 11381 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:16.725956 13096 consensus_queue.cc:1059] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20260321 02:59:16.745572 13306 raft_consensus.cc:2749] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:16.788760 13328 raft_consensus.cc:697] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:16.808455 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:16.811782 11972 raft_consensus.cc:2393] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 4: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
I20260321 02:59:16.775131 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:16.771128 11380 raft_consensus.cc:2393] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 5: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:16.815394 11377 raft_consensus.cc:2393] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 5: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:16.737982 13375 consensus_queue.cc:237] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.792395 11383 raft_consensus.cc:2393] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 5: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
W20260321 02:59:16.793210 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 117169us
I20260321 02:59:16.711648 13210 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:16.737977 13076 consensus_queue.cc:1059] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260321 02:59:16.772209 11376 raft_consensus.cc:1275] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260321 02:59:16.765605 13203 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:16.747849 13086 raft_consensus.cc:2749] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:16.760459 13272 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:16.802623 11378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40950 (request call id 1124) took 1074 ms. Trace:
I20260321 02:59:16.839001 11378 rpcz_store.cc:276] 0321 02:59:15.729043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:15.822201 (+ 93158us) service_pool.cc:225] Handling call
0321 02:59:15.840341 (+ 18140us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:16.580930 (+740589us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:16.802382 (+221452us) spinlock_profiling.cc:243] Waited 726 ms on lock 0x7b54001f8bd4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:16.802559 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1669082496}
I20260321 02:59:16.840543 13291 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:16.790380 13381 log.cc:927] Time spent T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.115s user 0.000s sys 0.001s
I20260321 02:59:16.741482 13253 raft_consensus.cc:697] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:16.786556 13315 raft_consensus.cc:2804] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:16.809495 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:16.810597 13389 raft_consensus.cc:3055] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:16.703359 13139 raft_consensus.cc:2749] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:16.813110 11379 raft_consensus.cc:2393] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 6: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:16.749181 12961 consensus_queue.cc:1059] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
W20260321 02:59:16.796700 13379 log.cc:927] Time spent T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.159s user 0.000s sys 0.001s
I20260321 02:59:16.811726 13328 consensus_queue.cc:237] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:16.818969 13378 log.cc:927] Time spent T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20260321 02:59:16.849745 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:16.872076 11381 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:16.821952 11200 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:16.804508 13383 raft_consensus.cc:3055] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:16.882652 13076 raft_consensus.cc:3055] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:16.883119 13076 raft_consensus.cc:740] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:16.883468 13337 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:16.878592 13383 raft_consensus.cc:740] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:16.883967 13076 consensus_queue.cc:260] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.884938 13383 consensus_queue.cc:260] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.885336 13076 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:16.886102 13383 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:16.886193 13339 raft_consensus.cc:3055] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Stepping down as leader of term 3
W20260321 02:59:16.886758 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:16.887168 13337 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:16.845364 13380 log.cc:927] Time spent T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.168s user 0.000s sys 0.001s
I20260321 02:59:16.801139 13353 raft_consensus.cc:2749] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:16.801528 13178 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:16.889355 13210 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:16.879168 13315 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:16.890151 13315 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:16.886703 13339 raft_consensus.cc:740] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:16.891475 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:16.889801 13178 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:16.794402 11349 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:16.791747 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 43819.6us
I20260321 02:59:16.820077 13096 raft_consensus.cc:3055] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:16.894260 11384 raft_consensus.cc:2393] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 6: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
W20260321 02:59:16.846184 13382 log.cc:927] Time spent T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.187s user 0.001s sys 0.000s
I20260321 02:59:16.899195 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:16.963074 11308 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 2.
I20260321 02:59:16.964260 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:16.901999 11349 leader_election.cc:336] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Timed out: RequestConsensusVote RPC to 127.9.213.1:36717 timed out after 1.528s (SENT)
I20260321 02:59:16.906493 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:16.960858 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:16.848233 13192 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:16.966050 11303 raft_consensus.cc:2376] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4. Re-sending same reply.
I20260321 02:59:16.961337 11275 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:16.966763 11304 raft_consensus.cc:2393] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 6: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
W20260321 02:59:16.967211 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 65265.9us
I20260321 02:59:16.880425 13373 raft_consensus.cc:3055] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:16.973328 11275 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:16.967399 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:16.968155 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:16.976815 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 154931us
I20260321 02:59:16.981101 11311 raft_consensus.cc:2393] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 4: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:16.873001 13389 raft_consensus.cc:740] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:16.983636 11275 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:16.889389 13337 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:16.954160 12961 consensus_queue.cc:1243] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer 83b12b883a7f4129a87dadf5a78f08da log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260321 02:59:16.988125 11349 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:16.971833 11972 raft_consensus.cc:2410] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260321 02:59:16.977897 11381 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:17.003643 11302 raft_consensus.cc:1240] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 6. Ops: [3.2-5.3]
W20260321 02:59:17.000214 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:17.004755 11349 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:16.962924 13272 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:17.005190 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:16.843840 13357 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:16.957659 13096 raft_consensus.cc:740] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:16.992892 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:17.006302 11349 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:17.007270 11275 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:16.974195 13260 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:16.981988 13353 raft_consensus.cc:2804] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:16.891764 13339 consensus_queue.cc:260] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:17.016603 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:16.889279 13253 consensus_queue.cc:237] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:17.020552 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:17.020884 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:17.011987 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:17.013648 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:17.021536 11232 raft_consensus.cc:2376] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4. Re-sending same reply.
I20260321 02:59:17.016152 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:17.017544 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:16.984251 13373 raft_consensus.cc:740] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:16.985966 13176 raft_consensus.cc:2804] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:16.993552 13301 raft_consensus.cc:2749] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:17.020628 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:17.024744 11229 raft_consensus.cc:2368] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 6.
I20260321 02:59:17.021312 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:17.021838 11230 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:17.013096 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:17.023196 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:17.025944 11234 raft_consensus.cc:1275] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260321 02:59:17.029109 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:17.029716 11236 raft_consensus.cc:2393] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 2: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
I20260321 02:59:17.009829 12961 raft_consensus.cc:2749] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:17.032666 11305 raft_consensus.cc:1240] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:17.030395 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:17.033713 11235 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:17.029803 11231 raft_consensus.cc:2468] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 2.
I20260321 02:59:17.008301 13355 raft_consensus.cc:2749] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:17.031349 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:17.002552 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:17.040725 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:17.033878 11275 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:17.030627 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:17.010260 13325 raft_consensus.cc:2804] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:17.017032 13353 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:17.038057 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:17.048491 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:17.050611 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:17.051251 13325 raft_consensus.cc:697] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:17.051777 11229 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:17.052354 13325 consensus_queue.cc:237] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:17.053552 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:17.055068 13242 consensus_queue.cc:1048] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:59:17.056226 13353 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:17.057778 11378 consensus_queue.cc:237] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: false } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:17.038371 13389 consensus_queue.cc:260] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:17.060995 13389 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:17.061906 11231 raft_consensus.cc:1275] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260321 02:59:17.063437 11231 raft_consensus.cc:1240] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:17.069259 11230 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:17.069713 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:17.070678 11234 raft_consensus.cc:2393] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 4: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:17.011734 13086 consensus_queue.cc:1048] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.028s
I20260321 02:59:17.082304 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:17.030161 13210 raft_consensus.cc:2749] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:17.078553 11384 raft_consensus.cc:3055] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:17.093739 11384 raft_consensus.cc:740] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:17.033223 13327 raft_consensus.cc:2804] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:17.090440 11228 raft_consensus.cc:3055] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:17.105962 11228 raft_consensus.cc:740] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:17.090935 11231 raft_consensus.cc:1240] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260321 02:59:17.095788 11227 raft_consensus.cc:1240] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260321 02:59:17.103300 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:17.015228 13272 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:17.094475 11384 consensus_queue.cc:260] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:17.047016 13176 raft_consensus.cc:697] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:17.111300 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:17.106805 11228 consensus_queue.cc:260] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:17.114293 11228 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:17.104007 11275 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:17.116488 13210 raft_consensus.cc:2804] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:17.117136 13210 raft_consensus.cc:697] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:17.118108 13210 consensus_queue.cc:237] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:17.126667 11233 raft_consensus.cc:1275] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260321 02:59:17.023144 13260 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:17.144698 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:17.129684 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 123475us
I20260321 02:59:17.150600 11349 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:17.126144 11942 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:17.069068 13301 raft_consensus.cc:2749] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:17.153417 11307 raft_consensus.cc:1275] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260321 02:59:17.153468 11232 raft_consensus.cc:1275] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260321 02:59:17.154673 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:17.155148 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:17.152057 11942 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:17.158300 11942 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:17.129258 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:17.128980 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:17.146234 11275 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:17.165561 11275 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:17.167181 11305 raft_consensus.cc:1240] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:17.173250 11305 raft_consensus.cc:1240] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:17.125514 13327 raft_consensus.cc:697] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 6 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:17.183912 11306 raft_consensus.cc:1240] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:17.184182 13235 raft_consensus.cc:2804] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:17.175601 13194 raft_consensus.cc:2804] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:17.186939 13194 raft_consensus.cc:697] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:17.177546 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:17.178747 11305 raft_consensus.cc:1240] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:17.188663 13194 consensus_queue.cc:237] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:17.190056 11233 raft_consensus.cc:2393] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:17.194972 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:17.197950 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:17.199541 11228 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:17.201540 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:17.201910 11303 raft_consensus.cc:1240] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:17.202430 11377 raft_consensus.cc:2393] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 4: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:17.203141 11304 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da"
dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:56630
I20260321 02:59:17.203737 11304 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:59:17.204128 11304 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:17.206144 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:17.110594 13361 consensus_queue.cc:1048] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20260321 02:59:17.209195 11382 raft_consensus.cc:1240] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:17.104568 13331 consensus_queue.cc:1059] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20260321 02:59:17.209893 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:17.218022 11305 raft_consensus.cc:1240] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:17.218612 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:17.219726 11311 raft_consensus.cc:1240] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:17.108264 13264 consensus_queue.cc:1059] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20260321 02:59:17.219739 11275 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:17.217082 11302 raft_consensus.cc:1240] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:17.080149 13096 consensus_queue.cc:260] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:17.162216 13360 raft_consensus.cc:2804] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260321 02:59:17.240911 11275 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:17.233048 11304 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:17.258370 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:17.203434 12976 raft_consensus.cc:2749] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:17.154599 13139 consensus_queue.cc:1048] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
I20260321 02:59:17.273913 11304 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:17.093469 13373 consensus_queue.cc:260] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:17.209280 13235 raft_consensus.cc:697] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:17.216635 13301 raft_consensus.cc:2749] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:17.283143 11383 raft_consensus.cc:1240] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7. Ops: [4.2-6.3]
I20260321 02:59:17.287693 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:17.325342 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 145058us
I20260321 02:59:17.341316 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:17.260257 13360 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:17.561388 11377 raft_consensus.cc:1240] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: []
W20260321 02:59:17.562306 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:17.376932 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 220514us
I20260321 02:59:17.272426 13355 consensus_queue.cc:1048] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.108s
I20260321 02:59:17.338205 13357 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:17.402981 13400 log.cc:927] Time spent T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.250s user 0.000s sys 0.000s
W20260321 02:59:17.431300 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 170254us
I20260321 02:59:17.566864 11275 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:17.568089 11275 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:17.285858 13192 consensus_queue.cc:1059] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.095s
I20260321 02:59:17.394651 13327 consensus_queue.cc:237] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:17.337481 13401 consensus_queue.cc:1059] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.107s
I20260321 02:59:17.261473 13176 consensus_queue.cc:237] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:17.285616 13375 consensus_queue.cc:1059] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.095s
I20260321 02:59:17.259877 12961 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:17.262966 13337 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:17.261234 13402 raft_consensus.cc:2804] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:17.546408 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:17.620445 11230 raft_consensus.cc:2468] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:17.549460 11304 raft_consensus.cc:1240] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:17.559612 11380 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:17.564445 11349 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:17.561959 11942 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:17.543906 13203 consensus_meta.cc:287] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.712s user 0.012s sys 0.001s
I20260321 02:59:17.573848 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:17.560109 13310 consensus_meta.cc:287] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.813s user 0.011s sys 0.005s
W20260321 02:59:17.613334 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 334147us
I20260321 02:59:17.616755 11377 raft_consensus.cc:1240] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
W20260321 02:59:17.617569 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:17.566056 13360 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 1 FOLLOWER]: Advancing to term 2
W20260321 02:59:17.611800 13398 log.cc:927] Time spent T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.360s user 0.000s sys 0.014s
I20260321 02:59:17.560454 13226 raft_consensus.cc:2749] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:17.562925 13264 raft_consensus.cc:2749] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260321 02:59:17.571359 13383 consensus_meta.cc:287] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.685s user 0.010s sys 0.008s
I20260321 02:59:17.630049 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:17.560719 13260 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:17.577339 13373 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
W20260321 02:59:17.972429 13403 log.cc:927] Time spent T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.173s user 0.000s sys 0.001s
W20260321 02:59:17.973733 13406 log.cc:927] Time spent T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.310s user 0.000s sys 0.001s
I20260321 02:59:17.973950 13096 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:17.664341 13412 raft_consensus.cc:2749] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260321 02:59:17.689980 13315 consensus_meta.cc:287] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 0.799s user 0.015s sys 0.000s
W20260321 02:59:17.663849 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 101963us
I20260321 02:59:17.635296 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:17.979943 11942 leader_election.cc:304] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:17.675088 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:17.981863 11380 raft_consensus.cc:2393] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 5: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
W20260321 02:59:17.746498 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 182114us
I20260321 02:59:17.985837 11349 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:17.962864 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:17.988644 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:17.989087 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:17.987465 11200 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:17.544574 12720 raft_consensus.cc:3055] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Stepping down as leader of term 5
W20260321 02:59:17.821625 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:18.012197 11376 raft_consensus.cc:1240] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: []
W20260321 02:59:17.704613 13405 log.cc:927] Time spent T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.130s user 0.000s sys 0.000s
W20260321 02:59:17.698745 13353 consensus_meta.cc:287] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.642s user 0.008s sys 0.008s
W20260321 02:59:17.693619 13307 consensus_meta.cc:287] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.894s user 0.009s sys 0.006s
W20260321 02:59:17.694046 13084 consensus_meta.cc:287] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.992s user 0.015s sys 0.002s
I20260321 02:59:18.018360 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:17.907109 13413 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:17.693630 13076 consensus_meta.cc:287] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.803s user 0.011s sys 0.002s
I20260321 02:59:18.029382 11380 raft_consensus.cc:1240] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:17.691228 13291 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:17.799355 13139 raft_consensus.cc:493] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:18.009974 13310 raft_consensus.cc:2749] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260321 02:59:17.991246 13397 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.648s user 0.008s sys 0.016s
I20260321 02:59:17.737478 13272 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:17.687278 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:17.619604 11972 raft_consensus.cc:1275] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260321 02:59:17.956576 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:18.050851 13307 raft_consensus.cc:2749] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:17.996325 13402 raft_consensus.cc:697] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:17.576524 13301 consensus_queue.cc:1059] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.453s
I20260321 02:59:17.571727 13388 consensus_queue.cc:1059] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.457s
I20260321 02:59:17.991983 13220 raft_consensus.cc:2804] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader election won for term 2
I20260321 02:59:17.991778 13384 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:17.736037 13416 raft_consensus.cc:2749] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:18.050956 11230 raft_consensus.cc:3055] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:17.907289 12976 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:17.624022 12961 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:17.699806 13235 consensus_queue.cc:237] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:17.629439 13337 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:18.045185 13401 raft_consensus.cc:3055] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:17.636299 13203 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.078881 11381 raft_consensus.cc:1240] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:18.081867 11234 raft_consensus.cc:2468] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
W20260321 02:59:18.084367 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:18.090008 11231 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:17.984347 13315 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:18.014499 12720 raft_consensus.cc:740] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
W20260321 02:59:18.103428 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:18.131790 13176 consensus_queue.cc:1059] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20260321 02:59:18.160889 13383 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:18.083906 13389 consensus_meta.cc:287] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 1.022s user 0.017s sys 0.001s
W20260321 02:59:18.084170 13216 consensus_meta.cc:287] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 1.343s user 0.012s sys 0.005s
I20260321 02:59:18.098187 13134 raft_consensus.cc:2749] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:18.138006 11309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49786 (request call id 162) took 1231 ms. Trace:
I20260321 02:59:18.206248 11236 raft_consensus.cc:2393] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 4: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:18.048558 13353 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.058954 13375 consensus_queue.cc:1059] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20260321 02:59:18.073144 13429 raft_consensus.cc:2749] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:18.150048 11230 raft_consensus.cc:740] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:18.213701 13437 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:18.214314 13437 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:18.214825 13203 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:18.214813 11230 consensus_queue.cc:260] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:18.150774 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:18.107442 13358 raft_consensus.cc:2749] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:18.108031 13421 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:18.216902 13421 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.067080 13291 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.218740 13315 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:18.072865 13139 raft_consensus.cc:515] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:18.220490 13178 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:18.220954 13291 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:18.221537 13139 leader_election.cc:290] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:18.164400 13220 raft_consensus.cc:697] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:18.222047 12961 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:18.222108 12720 consensus_queue.cc:260] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.222654 13220 consensus_queue.cc:237] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:18.223459 12720 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:18.164987 12976 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:18.065498 13413 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.227746 12976 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:18.228149 13413 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:18.150450 13388 consensus_queue.cc:1243] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer 83b12b883a7f4129a87dadf5a78f08da log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260321 02:59:18.230588 13353 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:18.176913 11229 raft_consensus.cc:606] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:18.231607 11229 raft_consensus.cc:614] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260321 02:59:18.079123 13397 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.740s user 0.008s sys 0.016s
I20260321 02:59:18.172866 13310 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:18.233044 13310 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:18.234599 13337 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:18.213186 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 201102us
I20260321 02:59:18.170378 13307 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:18.238269 13307 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.240450 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:18.240840 13307 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:18.241911 13360 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:18.212971 13389 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:18.247768 13389 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:18.250530 13389 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:18.222759 13134 raft_consensus.cc:3055] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:18.235500 13310 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:18.172092 13401 raft_consensus.cc:740] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:18.251241 13134 raft_consensus.cc:740] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:18.251648 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:18.252303 13401 consensus_queue.cc:260] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.202349 13439 raft_consensus.cc:2749] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:18.252408 13134 consensus_queue.cc:260] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:18.253573 13401 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:18.111045 13418 consensus_queue.cc:1059] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.087s
I20260321 02:59:18.254652 13418 raft_consensus.cc:3055] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:18.255790 13418 raft_consensus.cc:740] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:18.256871 13418 consensus_queue.cc:260] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.258306 11378 raft_consensus.cc:1240] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:18.258899 11381 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1468) took 1040 ms. Trace:
I20260321 02:59:18.218845 13421 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:18.259274 11381 rpcz_store.cc:276] 0321 02:59:17.218750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:17.219091 (+ 341us) service_pool.cc:225] Handling call
0321 02:59:17.219396 (+ 305us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:18.176268 (+956872us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:18.258656 (+ 82388us) spinlock_profiling.cc:243] Waited 859 ms on lock 0x7b540024bdd4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:18.258843 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1975984512}
I20260321 02:59:18.260600 11378 raft_consensus.cc:1275] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260321 02:59:18.263116 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:18.265429 13389 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:18.265717 11234 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58200 (request call id 1557) took 1190 ms. Trace:
I20260321 02:59:18.266199 13389 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.264710 13360 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:18.231765 11306 raft_consensus.cc:2393] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
W20260321 02:59:18.245788 13410 log.cc:927] Time spent T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.215s user 0.001s sys 0.000s
W20260321 02:59:18.246754 13417 log.cc:927] Time spent T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260321 02:59:18.206358 11309 rpcz_store.cc:276] 0321 02:59:16.906325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:16.907997 (+ 1672us) service_pool.cc:225] Handling call
0321 02:59:17.080810 (+172813us) spinlock_profiling.cc:243] Waited 7.75 ms on lock 0x7b5800002a98. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b1522680 00007fa8b150cfe9 00007fa8b150ca92 00007fa8aaa8ad34
0321 02:59:17.081012 (+ 202us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:17.091190 (+ 10178us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:17.091221 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:17.091716 (+ 495us) log.cc:844] Serialized 26 byte log entry
0321 02:59:17.220132 (+128416us) raft_consensus.cc:1645] Marking committed up to 6
0321 02:59:17.611829 (+391697us) log.cc:844] Serialized 14 byte log entry
0321 02:59:17.616890 (+ 5061us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:17.617064 (+ 174us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:18.129655 (+512591us) raft_consensus.cc:1685] finished
0321 02:59:18.137731 (+ 8076us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:18.137923 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57323,"prepare.queue_time_us":645643,"raft.queue_time_us":145610,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":66718,"spinlock_wait_cycles":29512192,"thread_start_us":442402,"threads_started":2,"wal-append.queue_time_us":269214}
W20260321 02:59:18.244801 13419 log.cc:927] Time spent T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
W20260321 02:59:18.176556 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:18.278620 11377 raft_consensus.cc:1275] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260321 02:59:18.279510 13373 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:18.280275 13373 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:18.281160 11233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58214 (request call id 1177) took 1134 ms. Trace:
I20260321 02:59:18.281776 13428 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:18.255110 11232 raft_consensus.cc:2468] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:18.266304 11234 rpcz_store.cc:276] 0321 02:59:17.075670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:17.076761 (+ 1091us) service_pool.cc:225] Handling call
0321 02:59:18.265513 (+1188752us) spinlock_profiling.cc:243] Waited 967 ms on lock 0x7b540019d7d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:18.265649 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2223736064}
I20260321 02:59:18.282876 13373 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:18.281519 11233 rpcz_store.cc:276] 0321 02:59:17.146780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:17.190823 (+ 44043us) service_pool.cc:225] Handling call
0321 02:59:17.191230 (+ 407us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:17.191655 (+ 425us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:17.191683 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:17.196022 (+ 4339us) log.cc:844] Serialized 26 byte log entry
0321 02:59:17.356576 (+160554us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:17.356635 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:17.356839 (+ 204us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:18.280965 (+924126us) raft_consensus.cc:1685] finished
0321 02:59:18.280988 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:18.281102 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":122,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":126,"thread_start_us":159772,"threads_started":1,"wal-append.queue_time_us":713632}
I20260321 02:59:18.283320 13431 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:18.283843 13431 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:18.152303 13384 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.285195 11200 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:18.286456 13431 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:18.286906 11200 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:18.287405 13420 raft_consensus.cc:3055] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:18.287797 13420 raft_consensus.cc:740] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:18.273125 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:18.288301 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:18.288475 11378 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:18.283746 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:18.289768 11234 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:18.290282 11234 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59202 (request call id 160) took 1034 ms. Trace:
I20260321 02:59:18.290196 11228 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59202 (request call id 156) took 1035 ms. Trace:
I20260321 02:59:18.290752 11234 rpcz_store.cc:276] 0321 02:59:17.256190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:18.283485 (+1027295us) service_pool.cc:225] Handling call
0321 02:59:18.290224 (+ 6739us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:18.290891 11228 rpcz_store.cc:276] 0321 02:59:17.254617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:17.254793 (+ 176us) service_pool.cc:225] Handling call
0321 02:59:17.255142 (+ 349us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:17.264375 (+ 9233us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:17.264423 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:17.276253 (+ 11830us) log.cc:844] Serialized 26 byte log entry
0321 02:59:17.621069 (+344816us) raft_consensus.cc:1645] Marking committed up to 6
0321 02:59:17.621109 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:17.621172 (+ 63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:18.289992 (+668820us) raft_consensus.cc:1685] finished
0321 02:59:18.290016 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:18.290134 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8726,"raft.queue_time_us":10105,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":13532,"thread_start_us":297639,"threads_started":1,"wal-append.queue_time_us":866535}
I20260321 02:59:18.282500 13428 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.295085 11233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59202 (request call id 157) took 1040 ms. Trace:
I20260321 02:59:18.286556 13384 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:18.295461 11385 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
W20260321 02:59:18.516384 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:18.516903 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:18.518100 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 221818us
I20260321 02:59:18.299706 11233 rpcz_store.cc:276] 0321 02:59:17.255008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:18.283670 (+1028662us) service_pool.cc:225] Handling call
0321 02:59:18.283943 (+ 273us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:18.294858 (+ 10915us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:18.294875 (+ 17us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:18.294904 (+ 29us) raft_consensus.cc:1645] Marking committed up to 5
0321 02:59:18.294926 (+ 22us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:18.294963 (+ 37us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:18.295043 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:18.333334 11234 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59202 (request call id 158) took 1078 ms. Trace:
I20260321 02:59:18.532362 11231 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58200 (request call id 1559) took 1440 ms. Trace:
I20260321 02:59:18.534371 11235 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58200 (request call id 1544) took 1517 ms. Trace:
W20260321 02:59:18.541298 11232 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58200 (request call id 1562) took 1389 ms (client timeout 1672 ms). Trace:
W20260321 02:59:18.591323 11232 rpcz_store.cc:269] 0321 02:59:17.151307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:17.154899 (+ 3592us) service_pool.cc:225] Handling call
0321 02:59:18.540741 (+1385842us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540037bc54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:18.541058 (+ 317us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2417000320}
I20260321 02:59:18.592537 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:18.543231 11236 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58200 (request call id 1543) took 1527 ms. Trace:
W20260321 02:59:18.325043 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:18.544991 11349 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.004s sys 0.023s
I20260321 02:59:18.594383 11236 rpcz_store.cc:276] 0321 02:59:17.016056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:17.030406 (+ 14350us) service_pool.cc:225] Handling call
0321 02:59:18.542676 (+1512270us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b540037b254. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:18.543010 (+ 334us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2674288896}
W20260321 02:59:18.595579 11349 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.580s user 0.004s sys 0.023s
W20260321 02:59:18.347321 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:18.553773 11306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56630 (request call id 1477) took 1360 ms. Trace:
I20260321 02:59:18.555012 11229 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:41224 (request call id 19) took 1543 ms. Trace:
I20260321 02:59:18.568430 11384 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:18.280269 11975 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
I20260321 02:59:18.372392 13123 raft_consensus.cc:3055] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:18.616585 11200 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:18.617365 13123 raft_consensus.cc:740] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:18.212639 13216 raft_consensus.cc:2749] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:18.430832 13242 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:18.437009 13328 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.094s user 0.000s sys 0.026s
I20260321 02:59:18.619649 13418 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:18.619853 13425 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:18.620460 13328 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.281s user 0.001s sys 0.026s
I20260321 02:59:18.274106 13337 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:18.621176 13425 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.622385 13337 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.205129 13383 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.624958 11972 raft_consensus.cc:1275] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260321 02:59:18.625200 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:18.625928 13444 consensus_queue.cc:1175] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "fef7f8cc55b043039131bd9b2229f0a5" responder_term: 3 status { last_received { term: 3 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 3 index: 2 } }
I20260321 02:59:18.626729 13383 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:18.378574 13389 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:18.334802 13388 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:18.206998 13402 consensus_queue.cc:237] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.627563 13446 consensus_queue.cc:1048] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260321 02:59:18.628994 13388 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.630529 11232 raft_consensus.cc:1275] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260321 02:59:18.382720 13399 log.cc:927] Time spent T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 1.173s user 0.004s sys 0.002s
W20260321 02:59:18.468261 13404 log.cc:927] Time spent T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.924s user 0.003s sys 0.004s
I20260321 02:59:18.288627 13420 consensus_queue.cc:260] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.635710 13420 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:18.383404 13427 raft_consensus.cc:2804] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:18.641114 13427 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:18.641817 13427 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:18.392882 13423 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:18.645262 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 659514us
I20260321 02:59:18.645095 13423 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.342448 13424 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:18.381387 13426 consensus_queue.cc:1059] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.085s
W20260321 02:59:18.373704 13432 log.cc:927] Time spent T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20260321 02:59:18.448602 13437 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:18.359547 13442 raft_consensus.cc:2749] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260321 02:59:18.660668 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 295437us
I20260321 02:59:18.374603 13210 raft_consensus.cc:3055] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:18.663735 13210 raft_consensus.cc:740] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:18.602813 11306 rpcz_store.cc:276] 0321 02:59:17.193192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:17.197535 (+ 4343us) service_pool.cc:225] Handling call
0321 02:59:18.553115 (+1355580us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540018b354. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:18.553466 (+ 351us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2373287680}
I20260321 02:59:18.665539 11378 raft_consensus.cc:1275] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260321 02:59:18.608094 11385 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40950 (request call id 1137) took 1817 ms (client timeout 1628 ms). Trace:
I20260321 02:59:18.591132 11231 rpcz_store.cc:276] 0321 02:59:17.091968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:17.107880 (+ 15912us) service_pool.cc:225] Handling call
0321 02:59:18.531613 (+1423733us) spinlock_profiling.cc:243] Waited 820 ms on lock 0x7b5400060754. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:18.532018 (+ 405us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1885802368}
I20260321 02:59:18.603165 11229 rpcz_store.cc:276] 0321 02:59:17.011543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:17.051597 (+ 40054us) service_pool.cc:225] Handling call
0321 02:59:18.554506 (+1502909us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400060754. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac546c33 00007fa8b1510cca 00007fa8aaa9d934 00007fa8aaa9d8c2 00007fa8aaa9d7f1 00007fa8aaa9d76c 00007fa8aaa9ca22 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:18.554744 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2585502976}
I20260321 02:59:18.582705 13390 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:18.615414 11234 rpcz_store.cc:276] 0321 02:59:17.255305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:18.291536 (+1036231us) service_pool.cc:225] Handling call
0321 02:59:18.332627 (+ 41091us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:18.332892 (+ 265us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:18.332930 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:18.332989 (+ 59us) raft_consensus.cc:1645] Marking committed up to 6
0321 02:59:18.333033 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:18.333106 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:18.333259 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:18.635851 11382 raft_consensus.cc:1240] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 6 LEADER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:18.431401 13428 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:18.653023 13424 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:18.591177 11235 rpcz_store.cc:276] 0321 02:59:17.016320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:17.034509 (+ 18189us) service_pool.cc:225] Handling call
0321 02:59:18.533821 (+1499312us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400379954. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:18.534147 (+ 326us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2446818048}
I20260321 02:59:18.666078 13412 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:18.613364 13192 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.973s user 0.009s sys 0.004s
I20260321 02:59:18.669584 13235 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
W20260321 02:59:18.671555 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:18.666599 11385 rpcz_store.cc:269] 0321 02:59:16.765451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:16.765586 (+ 135us) service_pool.cc:225] Handling call
0321 02:59:18.570252 (+1804666us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b540030eb54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:18.582925 (+ 12673us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3063822208}
W20260321 02:59:18.693748 13192 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.081s user 0.010s sys 0.005s
I20260321 02:59:18.702370 13235 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:18.712304 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:18.712752 13390 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.714262 13384 consensus_queue.cc:1059] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:59:18.716121 13331 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:18.716813 13412 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.717031 13331 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.718778 13444 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:18.726958 13444 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.727623 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:18.704342 13383 raft_consensus.cc:2749] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:18.729619 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:18.676453 13388 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:18.852330 11232 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58214 (request call id 1181) took 1277 ms. Trace:
I20260321 02:59:18.815162 11385 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:18.889123 11232 rpcz_store.cc:276] 0321 02:59:17.575308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:18.593010 (+1017702us) service_pool.cc:225] Handling call
0321 02:59:18.594031 (+ 1021us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:18.631458 (+ 37427us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:18.852045 (+220587us) spinlock_profiling.cc:243] Waited 36.1 ms on lock 0x7b54002f94d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:18.852231 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":82931968}
W20260321 02:59:18.825117 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 85556us
I20260321 02:59:18.850827 13396 raft_consensus.cc:3055] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:18.903182 13396 raft_consensus.cc:740] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:18.815382 13331 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:18.905686 11942 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:18.786175 13096 consensus_meta.cc:287] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.811s user 0.008s sys 0.014s
I20260321 02:59:18.677250 13242 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:18.907650 13427 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.826573 13301 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:18.909654 13301 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:18.910163 13242 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:18.910897 13331 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:18.911684 13331 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:18.912622 11384 raft_consensus.cc:1275] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260321 02:59:18.912431 13428 consensus_queue.cc:1048] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260321 02:59:18.912968 11384 pending_rounds.cc:85] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5: Aborting all ops after (but not including) 2
I20260321 02:59:18.913239 11384 pending_rounds.cc:107] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260321 02:59:18.913504 11384 raft_consensus.cc:2889] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:18.910269 13427 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:18.889671 11235 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
W20260321 02:59:18.730973 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 126467us
W20260321 02:59:18.858239 13447 log.cc:927] Time spent T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
I20260321 02:59:18.784664 13425 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:18.815383 13412 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:18.927908 11384 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40950 (request call id 1143) took 1945 ms. Trace:
I20260321 02:59:18.914045 13331 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:18.929376 11976 raft_consensus.cc:1275] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260321 02:59:18.904839 13396 consensus_queue.cc:260] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:18.930514 13138 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260321 02:59:18.792490 13337 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:18.792737 13390 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:18.730289 13423 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:18.790789 13444 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:18.931797 11235 raft_consensus.cc:606] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:18.931744 11384 rpcz_store.cc:276] 0321 02:59:16.975271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:16.975428 (+ 157us) service_pool.cc:225] Handling call
0321 02:59:16.975733 (+ 305us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:17.078477 (+102744us) spinlock_profiling.cc:243] Waited 26.1 ms on lock 0x7b54002abfa0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac4c41a0 00007fa8ac54e5a0 00007fa8ac54f611 00007fa8ac54c16f 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c
0321 02:59:18.567325 (+1488848us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b58008e04d0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac4a848a 00007fa8ac52a827 00007fa8ac5433c7 00007fa8ac544bbe 00007fa8ac54ef29 00007fa8ac54f7f6 00007fa8ac54c16f 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34
0321 02:59:18.914064 (+346739us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:18.920560 (+ 6496us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2760792320}
I20260321 02:59:18.946391 11235 raft_consensus.cc:614] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260321 02:59:18.953775 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:18.932030 13216 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
W20260321 02:59:18.959156 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 51558.1us
I20260321 02:59:18.864308 13306 consensus_queue.cc:1059] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.084s
W20260321 02:59:18.964200 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:18.964586 11200 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:18.961776 12720 consensus_meta.cc:287] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.710s user 0.010s sys 0.011s
W20260321 02:59:18.967180 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:18.965649 13401 consensus_meta.cc:287] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.608s user 0.016s sys 0.005s
I20260321 02:59:18.976089 11306 raft_consensus.cc:1240] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:18.983620 11235 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:18.964282 13457 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:18.965279 13423 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:18.970863 13322 raft_consensus.cc:2804] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:18.996306 13457 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:18.997759 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:18.984328 13396 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
W20260321 02:59:18.999279 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:18.999439 13412 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:18.999918 13390 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:19.001240 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:19.002339 11306 raft_consensus.cc:2393] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 4: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:19.001890 13390 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:19.002810 12720 raft_consensus.cc:2749] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:19.003849 13412 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:19.004529 13390 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:19.006168 13412 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:18.972460 13424 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:18.973663 13216 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:19.009033 13216 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:18.994007 13301 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:19.016251 13306 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:19.020612 13306 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.026216 11309 raft_consensus.cc:1275] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260321 02:59:19.026849 11235 raft_consensus.cc:606] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:19.028060 11235 raft_consensus.cc:614] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:19.028018 11227 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:19.029047 13375 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:19.029551 13375 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:19.033370 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.034727 13457 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:18.993911 13418 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:19.034773 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:19.035882 11972 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:19.041960 11309 raft_consensus.cc:1275] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260321 02:59:19.043993 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 53526.1us
I20260321 02:59:19.045284 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:19.045850 11201 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:19.046494 13390 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:19.047104 13390 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.045977 11306 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:19.043725 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:19.048189 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 83210.3us
W20260321 02:59:19.026333 11349 leader_election.cc:336] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Timed out: RequestConsensusVote RPC to 127.9.213.1:36717 timed out after 1.877s (SENT)
I20260321 02:59:19.026968 13306 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:19.048966 11349 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:19.053153 11304 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 2.
W20260321 02:59:19.053570 13458 log.cc:927] Time spent T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.085s user 0.001s sys 0.001s
I20260321 02:59:19.035678 13418 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.053172 11228 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58214 (request call id 1178) took 1424 ms. Trace:
I20260321 02:59:19.061055 13446 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:19.061901 13242 consensus_queue.cc:1048] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:19.062644 13203 raft_consensus.cc:3055] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:19.063170 13203 raft_consensus.cc:740] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:19.061911 13446 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:19.064316 13203 consensus_queue.cc:260] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.065622 13203 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:19.066792 13446 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:19.068140 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:19.069414 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 126978us
I20260321 02:59:19.048152 13423 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.070374 13123 consensus_queue.cc:260] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.072130 13423 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:19.072830 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.073187 13390 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.073787 13301 consensus_queue.cc:1048] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:19.054203 13412 consensus_queue.cc:1048] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260321 02:59:19.054634 13216 consensus_queue.cc:1048] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.023s
I20260321 02:59:19.077663 13418 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.057721 11228 rpcz_store.cc:276] 0321 02:59:17.573944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:18.293052 (+719108us) service_pool.cc:225] Handling call
0321 02:59:18.556272 (+263220us) raft_consensus.cc:1471] Updating replica for 2 ops
0321 02:59:18.588275 (+ 32003us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:18.588329 (+ 54us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0321 02:59:18.772545 (+184216us) log.cc:844] Serialized 52 byte log entry
0321 02:59:18.965531 (+192986us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:18.965576 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:18.965653 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:18.997578 (+ 31925us) raft_consensus.cc:1685] finished
0321 02:59:18.997603 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:18.997778 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":115257,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":17319,"thread_start_us":143227,"threads_started":2,"wal-append.queue_time_us":157242}
I20260321 02:59:19.047896 13420 raft_consensus.cc:2804] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:19.079734 13139 consensus_queue.cc:1048] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.002s
W20260321 02:59:19.081305 11227 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58200 (request call id 1560) took 1972 ms (client timeout 1717 ms). Trace:
W20260321 02:59:19.082333 11227 rpcz_store.cc:269] 0321 02:59:17.108432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:17.110018 (+ 1586us) service_pool.cc:225] Handling call
0321 02:59:19.081103 (+1971085us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b540027f1d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:19.081235 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4131571584}
I20260321 02:59:19.075250 13123 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:19.083078 13420 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:19.083690 13420 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
W20260321 02:59:19.084738 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 58469.7us
W20260321 02:59:19.085546 11349 connection.cc:232] RPC call timeout handler was delayed by -1.7488876270000446s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16310000000000002s for a response
W20260321 02:59:19.086031 11349 connection.cc:232] RPC call timeout handler was delayed by -1.7353876270000228s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1646s for a response
I20260321 02:59:19.086361 13401 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:19.087294 13401 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.088487 13235 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:19.081357 13424 raft_consensus.cc:2749] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:19.088836 13210 consensus_queue.cc:260] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.090325 13210 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:19.089540 13401 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:19.086380 11349 connection.cc:232] RPC call timeout handler was delayed by -1.7146876269999893s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16690000000000002s for a response
I20260321 02:59:19.093626 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:19.094280 11349 connection.cc:232] RPC call timeout handler was delayed by -1.4707876270000497s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.194s for a response
I20260321 02:59:19.094480 11972 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
W20260321 02:59:19.094663 11349 connection.cc:232] RPC call timeout handler was delayed by -1.4581876270000294s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19540000000000002s for a response
W20260321 02:59:19.095005 11349 connection.cc:232] RPC call timeout handler was delayed by -1.4518876270000192s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19610000000000002s for a response
W20260321 02:59:19.095317 11349 connection.cc:232] RPC call timeout handler was delayed by -1.4023876270000528s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2016s for a response
W20260321 02:59:19.095609 11349 connection.cc:232] RPC call timeout handler was delayed by -1.3870876270000281s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2033s for a response
I20260321 02:59:19.095522 13390 consensus_queue.cc:1048] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:19.096796 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:19.098721 11200 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:19.100740 13076 raft_consensus.cc:2804] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260321 02:59:19.101423 11349 connection.cc:232] RPC call timeout handler was delayed by -1.3762876270000106s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20450000000000002s for a response
I20260321 02:59:19.101643 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:19.102155 11349 connection.cc:232] RPC call timeout handler was delayed by -1.3438876270000719s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2081s for a response
W20260321 02:59:19.102483 11349 connection.cc:232] RPC call timeout handler was delayed by -1.3294876270000486s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2097s for a response
W20260321 02:59:19.102777 11349 connection.cc:232] RPC call timeout handler was delayed by -1.2979876269999977s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21320000000000003s for a response
W20260321 02:59:19.103055 11349 connection.cc:232] RPC call timeout handler was delayed by -1.2223876269999892s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22160000000000002s for a response
I20260321 02:59:19.103310 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:19.100674 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:19.101403 13076 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:19.103888 11349 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:19.126502 13235 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:19.128940 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:19.131026 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:19.120525 11201 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:19.130306 11382 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1469) took 1894 ms. Trace:
I20260321 02:59:19.116304 11235 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
W20260321 02:59:19.141851 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.128572 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:19.123167 11276 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:19.129756 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:19.128613 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:19.131142 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:19.145272 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 64312.3us
I20260321 02:59:19.142213 11235 raft_consensus.cc:606] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:19.104084 11972 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:19.148630 11229 raft_consensus.cc:1240] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260321 02:59:19.129767 13134 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:19.150651 13425 raft_consensus.cc:3055] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:19.151101 13425 raft_consensus.cc:740] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:19.152197 11384 raft_consensus.cc:2468] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 1.
I20260321 02:59:19.125808 11230 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:19.106841 13322 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:19.141742 11382 rpcz_store.cc:276] 0321 02:59:17.235544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:17.235881 (+ 337us) service_pool.cc:225] Handling call
0321 02:59:17.275486 (+ 39605us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:18.670107 (+1394621us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:19.130043 (+459936us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54000cd854. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:19.130246 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3126562688}
I20260321 02:59:19.121678 13076 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:19.104421 13084 raft_consensus.cc:3055] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:19.160869 11231 raft_consensus.cc:1240] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:19.162429 11276 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:19.161188 13123 raft_consensus.cc:2804] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:19.163379 13123 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:19.165673 13123 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:19.168989 13306 raft_consensus.cc:2749] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:19.169842 13216 raft_consensus.cc:2804] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:19.170020 11231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:19.170282 13216 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:19.170663 13216 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:19.171026 11229 raft_consensus.cc:1275] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 2. (index mismatch)
I20260321 02:59:19.170665 11231 raft_consensus.cc:606] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:19.172511 11231 raft_consensus.cc:614] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:19.172842 11232 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
W20260321 02:59:19.174957 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.172104 13291 raft_consensus.cc:2804] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:19.108592 13339 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:19.117471 13086 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:19.160946 13084 raft_consensus.cc:740] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:19.178640 13235 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.181816 13412 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:19.182566 13412 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.182675 13084 consensus_queue.cc:260] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.184188 13084 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:19.184283 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260321 02:59:19.184628 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.184702 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:19.185525 11231 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:19.183681 13235 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.184374 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:19.187744 13412 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.188643 13375 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.178990 13291 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:19.189677 13291 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:19.190495 11130 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44688 (request call id 129) took 9673 ms. Trace:
I20260321 02:59:19.190596 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:19.191443 11382 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:19.161768 13322 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:19.181218 11228 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:19.198402 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:19.198998 11235 raft_consensus.cc:1275] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260321 02:59:19.202497 11229 raft_consensus.cc:1240] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260321 02:59:19.202279 13461 consensus_queue.cc:1048] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260321 02:59:19.179307 13425 consensus_queue.cc:260] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.209062 13425 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:19.208784 13358 consensus_queue.cc:1048] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260321 02:59:19.211220 11385 raft_consensus.cc:1240] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:19.212363 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:19.213145 11383 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:19.211264 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:19.214303 11132 catalog_manager.cc:5671] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:59:19.213613 11385 raft_consensus.cc:1275] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260321 02:59:19.215940 11942 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:19.216480 11132 catalog_manager.cc:5671] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 3 to 5, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 5 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:19.222143 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:19.221840 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:19.221827 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:19.224006 11229 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:19.227623 11384 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:19.227785 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:19.229130 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:19.229161 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:19.233131 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.236732 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:19.234194 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:19.191114 11130 rpcz_store.cc:276] 0321 02:59:09.517099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:09.517466 (+ 367us) service_pool.cc:225] Handling call
0321 02:59:19.190239 (+9672773us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:59:16.422521 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:59:16.434638 (+ 12117us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:59:16.434677 (+ 39us) write_op.cc:435] Acquired schema lock
0321 02:59:16.434697 (+ 20us) tablet.cc:661] Decoding operations
0321 02:59:16.435167 (+ 470us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:59:16.435266 (+ 99us) write_op.cc:641] Partition lock acquired for write op
0321 02:59:16.435298 (+ 32us) tablet.cc:684] Acquiring locks for 2 operations
0321 02:59:16.435609 (+ 311us) tablet.cc:700] Row locks acquired
0321 02:59:16.435627 (+ 18us) write_op.cc:260] PREPARE: finished
0321 02:59:16.435778 (+ 151us) write_op.cc:270] Start()
0321 02:59:16.435907 (+ 129us) write_op.cc:276] Timestamp: P: 1774061956435729 usec, L: 0
0321 02:59:16.435928 (+ 21us) op_driver.cc:348] REPLICATION: starting
0321 02:59:16.536182 (+100254us) log.cc:844] Serialized 907 byte log entry
0321 02:59:17.260764 (+724582us) op_driver.cc:464] REPLICATION: finished
0321 02:59:18.615519 (+1354755us) write_op.cc:301] APPLY: starting
0321 02:59:18.615918 (+ 399us) tablet.cc:1366] starting BulkCheckPresence
0321 02:59:18.616236 (+ 318us) tablet.cc:1369] finished BulkCheckPresence
0321 02:59:18.616493 (+ 257us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:59:18.617270 (+ 777us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:59:18.618051 (+ 781us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0321 02:59:18.618151 (+ 100us) write_op.cc:312] APPLY: finished
0321 02:59:18.618731 (+ 580us) log.cc:844] Serialized 29 byte log entry
0321 02:59:18.699327 (+ 80596us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:18.766551 (+ 67224us) write_op.cc:454] Released schema lock
0321 02:59:18.777160 (+ 10609us) write_op.cc:341] FINISH: Updating metrics
0321 02:59:18.778170 (+ 1010us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:18.778227 (+ 57us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":462697,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":1004861,"apply.run_cpu_time_us":103750,"apply.run_wall_time_us":499618,"num_ops":2,"prepare.queue_time_us":280273,"prepare.run_cpu_time_us":3913,"prepare.run_wall_time_us":124152,"replication_time_us":736219,"spinlock_wait_cycles":6016,"thread_start_us":1172689,"threads_started":3,"wal-append.queue_time_us":291933}]]}
I20260321 02:59:19.243264 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:19.243978 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:19.244541 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:19.245093 13358 raft_consensus.cc:2804] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:19.245704 13358 raft_consensus.cc:697] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:19.246179 11302 raft_consensus.cc:1240] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:19.246807 13358 consensus_queue.cc:237] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:19.195246 13086 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.245127 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:19.248241 11303 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:19.248433 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:19.248705 11308 raft_consensus.cc:1240] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:19.244554 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:19.227653 13357 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:19.249986 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:19.249363 13086 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.250253 13357 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.251446 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:19.252393 13357 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.253129 11304 raft_consensus.cc:1217] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[4.2-6.3] Dedup: 4.2->[6.3-6.3]
I20260321 02:59:19.253681 11304 raft_consensus.cc:1240] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7. Ops: [4.2-6.3]
I20260321 02:59:19.239220 11378 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:19.244693 11377 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:19.255074 11385 raft_consensus.cc:2410] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260321 02:59:19.238503 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:19.247516 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:19.289347 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:19.260987 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 157207us
I20260321 02:59:19.324568 11382 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:19.365990 11232 raft_consensus.cc:2468] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:19.257339 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:19.360769 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:19.393769 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.289698 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:19.379732 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.324311 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:19.373652 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:19.397001 11230 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:19.397099 11304 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
W20260321 02:59:19.397505 11230 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58214 (request call id 1180) took 1822 ms (client timeout 1788 ms). Trace:
I20260321 02:59:19.304693 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:19.329097 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:59:19.330641 11309 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
W20260321 02:59:19.397909 11230 rpcz_store.cc:269] 0321 02:59:17.574920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:18.018010 (+443090us) service_pool.cc:225] Handling call
0321 02:59:18.553115 (+535105us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b18002ed5e0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac587a26 00007fa8ac4c196e 00007fa8ac54337e 00007fa8ac544bbe 00007fa8ac55065c 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c
0321 02:59:19.124884 (+571769us) spinlock_profiling.cc:243] Waited 569 ms on lock 0x7b58005105d0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac4a848a 00007fa8ac52a827 00007fa8ac5433c7 00007fa8ac544bbe 00007fa8ac55065c 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c
0321 02:59:19.397430 (+272546us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2017801344}
I20260321 02:59:19.403978 11235 raft_consensus.cc:1275] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260321 02:59:19.404886 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.408934 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:19.412407 13427 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:19.412737 13264 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:19.413081 13427 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.413398 13264 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.414695 13322 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:19.415598 13264 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.261302 13420 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:19.420877 11305 raft_consensus.cc:2393] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 6: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:19.416906 11306 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:19.415997 11977 raft_consensus.cc:1240] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260321 02:59:19.420388 11307 raft_consensus.cc:2410] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260321 02:59:19.402221 11310 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:19.415380 13427 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.376644 13357 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:19.368805 13301 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:19.473582 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:19.487610 13322 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:19.456013 13357 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:19.441156 13096 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:19.457594 11228 raft_consensus.cc:2393] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 5: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:19.448112 11942 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:19.465507 11132 catalog_manager.cc:5671] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:19.468226 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:19.472573 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:19.486977 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 84865.3us
W20260321 02:59:19.466760 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:19.450388 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.359046 13216 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:19.400131 13203 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:19.473763 11200 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:19.443246 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:19.532624 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:19.532972 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.441835 11275 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:19.324932 13123 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:19.367673 13235 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:19.486557 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:19.481899 13076 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.521447 13301 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.408525 13291 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:19.519552 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:19.493952 13412 consensus_queue.cc:1059] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260321 02:59:19.540527 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:19.529958 13459 raft_consensus.cc:2804] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:19.545572 11377 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:19.329347 13139 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:19.529088 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.547472 11385 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:19.526728 11378 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:19.548513 11311 raft_consensus.cc:2393] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:19.545442 13445 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:19.551204 13445 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:19.552645 11231 raft_consensus.cc:2393] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 7: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:19.553092 13420 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:19.554791 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:19.555387 11977 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:19.556207 13291 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:19.556310 13412 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election won for term 7
W20260321 02:59:19.556967 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.557713 13424 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:19.558295 13424 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.533517 13242 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:19.536494 13383 raft_consensus.cc:2749] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:19.564278 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:19.548233 13459 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:19.568817 11976 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Advancing to term 3
W20260321 02:59:19.572901 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 107504us
I20260321 02:59:19.560370 13412 raft_consensus.cc:697] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:19.556993 13331 raft_consensus.cc:3055] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:19.567262 11379 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:19.558795 13235 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.545899 13203 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:19.587883 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 186450us
I20260321 02:59:19.569283 13459 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:19.543692 13096 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:19.590044 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 50904.7us
I20260321 02:59:19.581594 13210 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:19.566381 13242 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.605020 11349 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:19.574626 13331 raft_consensus.cc:740] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
W20260321 02:59:19.609287 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 201574us
I20260321 02:59:19.574584 13139 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.629161 13291 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
W20260321 02:59:19.656402 13468 log.cc:927] Time spent T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20260321 02:59:19.657577 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:19.653458 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:19.629083 13339 raft_consensus.cc:2749] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260321 02:59:19.660387 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 55527.8us
I20260321 02:59:19.655923 13291 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:19.662135 11349 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:19.662312 13424 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:19.662767 13291 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:19.641155 13357 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:19.657874 13242 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.662921 13412 consensus_queue.cc:237] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.665788 11977 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:19.667203 11200 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:19.641409 11379 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40950 (request call id 1144) took 2661 ms (client timeout 2039 ms). Trace:
W20260321 02:59:19.668922 11379 rpcz_store.cc:269] 0321 02:59:16.975672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:16.988763 (+ 13091us) service_pool.cc:225] Handling call
0321 02:59:19.624295 (+2635532us) spinlock_profiling.cc:243] Waited 2.54 s on lock 0x7b54001d9f54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:19.637398 (+ 13103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5834757504}
I20260321 02:59:19.672580 11201 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:19.674053 11377 raft_consensus.cc:3060] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 1 FOLLOWER]: Advancing to term 2
I20260321 02:59:19.635248 13264 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:19.639663 13469 log.cc:927] Time spent T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260321 02:59:19.665061 13216 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:19.666941 13123 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:19.669130 13424 raft_consensus.cc:2749] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:19.640527 13210 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.673357 13076 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.692956 11382 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:19.683007 13383 raft_consensus.cc:2804] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:19.681288 13301 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.680295 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:19.676842 13084 raft_consensus.cc:2749] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:19.708818 11973 raft_consensus.cc:2393] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 5: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:19.708966 13427 consensus_queue.cc:1048] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260321 02:59:19.714411 13331 consensus_queue.cc:260] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:19.715185 11349 leader_election.cc:304] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:19.715788 13331 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:19.689225 13264 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.719285 13264 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.703084 13235 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:19.722451 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.723026 13096 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.706189 13291 consensus_queue.cc:1048] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.019s
I20260321 02:59:19.725719 13291 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:19.726152 13291 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.728940 13203 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.702639 13383 raft_consensus.cc:697] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:19.730733 13383 consensus_queue.cc:237] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:19.732942 13445 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:19.733960 13203 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:19.734488 13203 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.701996 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:19.736493 13203 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.737636 11275 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:19.738477 13235 raft_consensus.cc:2749] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:19.738974 11275 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:19.739734 13235 raft_consensus.cc:2804] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election won for term 5
W20260321 02:59:19.747020 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 55284.3us
I20260321 02:59:19.746464 11349 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:19.746387 13235 raft_consensus.cc:697] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
W20260321 02:59:19.750048 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.749938 13235 consensus_queue.cc:237] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:19.752019 13264 consensus_queue.cc:1048] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:59:19.757762 13210 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.748575 11976 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:19.760471 11201 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
W20260321 02:59:19.773406 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 62051.5us
I20260321 02:59:19.779677 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:19.727914 13291 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.780411 11976 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:19.782009 11307 raft_consensus.cc:1275] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260321 02:59:19.782284 11349 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:19.809278 11350 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:19.783778 11377 raft_consensus.cc:1275] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260321 02:59:19.773842 13096 raft_consensus.cc:2804] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election won for term 3
W20260321 02:59:19.791592 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.792007 13264 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:19.793568 11311 raft_consensus.cc:1240] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20260321 02:59:19.799474 13203 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:19.800901 13139 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.805809 13076 raft_consensus.cc:993] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb: : Instructing follower 83b12b883a7f4129a87dadf5a78f08da to start an election
I20260321 02:59:19.814795 13203 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:19.815642 13076 raft_consensus.cc:1081] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Signalling peer 83b12b883a7f4129a87dadf5a78f08da to start an election
I20260321 02:59:19.813066 13084 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:19.816993 13203 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:19.816840 13084 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:19.818918 13084 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.813805 13322 consensus_queue.cc:1048] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:59:19.810423 13401 raft_consensus.cc:2749] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260321 02:59:19.780164 13445 raft_consensus.cc:2764] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260321 02:59:19.818819 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:19.813657 13096 raft_consensus.cc:697] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:19.828140 13096 consensus_queue.cc:237] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:19.829031 11200 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:19.819569 11349 leader_election.cc:400] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 6.
I20260321 02:59:19.825892 13339 raft_consensus.cc:2804] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:19.822700 13424 raft_consensus.cc:2804] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:19.841621 11200 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:19.844874 11200 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:19.846074 11200 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:19.847431 11200 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:19.835589 13339 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:19.837776 13084 raft_consensus.cc:2804] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:19.837199 13424 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:19.856881 13339 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:19.857185 13424 raft_consensus.cc:3060] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:19.857118 13084 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:19.858042 13084 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:19.851150 13427 raft_consensus.cc:2804] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:19.859683 13427 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:19.850104 13203 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:19.860106 13427 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:19.860375 13203 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:19.860762 13203 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:19.862443 11349 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:19.850221 13076 raft_consensus.cc:2804] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:19.864126 13076 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:19.864209 13445 raft_consensus.cc:2749] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:19.864578 13076 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:19.854861 13134 raft_consensus.cc:2804] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:19.863814 11349 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:19.868186 13134 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:19.868824 13134 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:19.869354 13445 raft_consensus.cc:2749] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:19.870409 13084 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.839219 13264 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.821205 13459 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:19.877513 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.882230 13420 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:19.885032 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:19.888155 11972 raft_consensus.cc:1240] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:19.888753 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:19.891942 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:19.892313 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:19.892838 11231 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:19.894012 11228 raft_consensus.cc:2410] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260321 02:59:19.894507 11231 raft_consensus.cc:1240] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:19.895738 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:19.896392 11231 raft_consensus.cc:1275] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260321 02:59:19.896647 11229 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:19.901042 13306 consensus_queue.cc:1059] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260321 02:59:19.897611 13445 consensus_queue.cc:1048] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.007s
I20260321 02:59:19.885761 13420 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.906929 13424 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:19.908457 13420 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:19.910789 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.923671 13264 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.929899 13076 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:19.932340 13076 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.938318 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:19.936067 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:19.939237 11228 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:19.941566 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:19.945312 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.961509 13459 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.962970 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:19.964636 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:19.969483 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 57421us
I20260321 02:59:19.965299 11311 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:19.936900 13425 raft_consensus.cc:3055] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:19.973387 11230 raft_consensus.cc:1240] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:19.962868 13084 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.976709 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:19.977789 11311 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:19.976701 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:19.982421 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:19.991357 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.996886 13424 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:19.997967 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:19.999665 13134 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.000905 11229 raft_consensus.cc:1240] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:20.001770 13134 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:19.973268 13425 raft_consensus.cc:740] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:20.006428 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:20.017153 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:20.005683 11307 raft_consensus.cc:1275] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260321 02:59:20.014748 11303 raft_consensus.cc:1275] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260321 02:59:20.020375 11308 raft_consensus.cc:2393] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:20.003747 13425 consensus_queue.cc:260] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:20.016439 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:20.008806 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:20.021498 11302 raft_consensus.cc:2368] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 4. Current term is 5.
I20260321 02:59:20.021631 13425 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:20.028383 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:20.029129 11228 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:20.008458 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:20.032709 13203 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.033514 11349 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:20.035290 13203 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:20.037178 11227 raft_consensus.cc:1240] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:20.039074 11349 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:20.006027 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:20.042287 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.007982 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:20.044817 11377 raft_consensus.cc:2410] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260321 02:59:20.044831 11305 raft_consensus.cc:2393] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:20.007119 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:20.046687 11309 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
W20260321 02:59:20.046924 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:20.047780 11310 raft_consensus.cc:2468] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:20.048558 11235 raft_consensus.cc:1275] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260321 02:59:20.043053 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:20.056598 11229 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:20.057572 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:20.059252 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:20.059774 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.060004 11382 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:20.060509 11376 raft_consensus.cc:2468] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:20.061412 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.062711 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:20.063369 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.064218 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:20.064924 11307 raft_consensus.cc:2393] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:20.059414 11305 raft_consensus.cc:1240] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:20.065672 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:20.066239 11302 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:20.067162 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.067777 11377 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:20.068184 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:20.068725 11308 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:20.070199 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.072242 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:20.082020 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.083204 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:20.083590 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.083822 11310 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:20.085671 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.082944 11380 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:20.087069 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:20.043283 13401 raft_consensus.cc:2749] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:20.091272 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:20.049887 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:20.095525 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:20.095603 11383 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:20.100858 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:20.101104 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.102907 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:20.103507 11380 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
W20260321 02:59:20.104959 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 69589us
I20260321 02:59:20.101554 11303 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:20.084659 13424 raft_consensus.cc:2749] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:20.109530 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:20.110255 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:20.117717 11307 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:20.115006 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:20.118839 11303 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:20.119791 11229 raft_consensus.cc:2468] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
W20260321 02:59:20.121313 11200 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:20.121522 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:20.056000 13339 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:20.123729 11377 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:20.125249 11200 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:20.126528 13076 raft_consensus.cc:2749] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:20.126634 11200 leader_election.cc:304] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:20.132997 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:20.133724 11306 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:20.133939 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:20.135032 11309 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d"
dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:56630
I20260321 02:59:20.135746 11309 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:59:20.136237 11309 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:20.137332 11308 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
W20260321 02:59:20.137925 13485 log.cc:927] Time spent T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.100s user 0.001s sys 0.000s
I20260321 02:59:20.138998 11349 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:20.141090 13445 raft_consensus.cc:2749] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:20.147423 11302 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:20.148952 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:20.149916 11307 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:20.156350 13461 consensus_queue.cc:1059] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260321 02:59:20.157684 13461 raft_consensus.cc:3055] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:20.158103 13461 raft_consensus.cc:740] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:20.162781 11377 raft_consensus.cc:1275] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260321 02:59:20.165263 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:20.166625 11302 raft_consensus.cc:2410] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260321 02:59:20.174471 11349 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:20.179803 13427 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.182046 13427 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:20.182196 11309 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:20.182564 13331 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:20.213914 11302 raft_consensus.cc:1240] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:20.184751 11349 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:20.186554 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:20.186405 11382 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:20.216435 11380 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:20.213677 11303 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:20.187239 11307 raft_consensus.cc:1240] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:20.188071 13339 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:20.224395 13383 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:20.189536 11228 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 5.
I20260321 02:59:20.190515 13459 consensus_queue.cc:1048] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260321 02:59:20.225145 13383 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.191203 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:20.226678 11227 raft_consensus.cc:2393] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 7: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:20.192288 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:20.227989 11232 raft_consensus.cc:2393] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 6: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:20.159327 13461 consensus_queue.cc:260] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:20.194955 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:20.229014 11308 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:20.232337 11309 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:20.199177 11306 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:20.200430 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
W20260321 02:59:20.201472 11942 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.536s (SENT) [suppressed 34 similar messages]
I20260321 02:59:20.183033 13424 raft_consensus.cc:2804] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:20.233295 13445 raft_consensus.cc:2804] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election won for term 5
W20260321 02:59:20.249152 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 64497.6us
I20260321 02:59:20.249516 13445 raft_consensus.cc:697] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:20.250139 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:20.250726 11349 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:20.250521 13445 consensus_queue.cc:237] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.221220 13331 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.252337 13401 raft_consensus.cc:2804] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:20.252655 11975 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:20.252962 13401 raft_consensus.cc:697] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:20.254001 13401 consensus_queue.cc:237] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.254441 13331 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:20.250862 11972 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:20.221391 13076 raft_consensus.cc:2804] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260321 02:59:20.227887 13383 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:20.265024 11349 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:20.265986 11308 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:20.266317 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:20.272812 11228 raft_consensus.cc:1275] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260321 02:59:20.276633 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 75196.5us
I20260321 02:59:20.263397 13076 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:20.280756 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 154208us
I20260321 02:59:20.263870 13424 raft_consensus.cc:697] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:20.272511 13084 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:20.287490 11200 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:20.296022 11200 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:20.296438 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:20.291292 13424 consensus_queue.cc:237] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.298197 11942 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:20.282775 13076 raft_consensus.cc:3060] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:20.297255 11228 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:20.298444 11200 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:20.314363 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 82878us
I20260321 02:59:20.317229 11200 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:20.305497 13203 raft_consensus.cc:2804] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:20.318672 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:20.320914 11200 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:20.307333 13427 raft_consensus.cc:2804] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:20.322045 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:20.306634 13358 consensus_queue.cc:1048] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.014s
I20260321 02:59:20.324137 11942 leader_election.cc:400] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Vote denied by peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267) with higher term. Message: Invalid argument: T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 4. Current term is 5.
I20260321 02:59:20.328486 11228 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:20.312952 13412 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:20.330801 11200 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:20.333070 11942 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:20.334072 13291 raft_consensus.cc:2804] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:20.334512 13291 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:20.334637 11200 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:20.334981 13291 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:20.314833 13446 raft_consensus.cc:2749] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:20.320593 13383 raft_consensus.cc:2804] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:20.318547 13306 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:20.308609 13084 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.322819 13134 raft_consensus.cc:2749] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:20.308853 13096 consensus_queue.cc:1048] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260321 02:59:20.362648 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:20.321041 13203 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:20.323575 13427 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:20.328358 13420 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:20.330462 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:20.363867 11200 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:20.363569 11231 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
W20260321 02:59:20.370757 11275 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:20.368844 11233 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:20.358247 13306 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:20.351274 13412 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.377804 13306 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:20.380064 13412 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:20.330132 13425 consensus_queue.cc:1059] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260321 02:59:20.381700 13425 raft_consensus.cc:3055] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:20.382184 13425 raft_consensus.cc:740] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:20.383502 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:20.367580 13203 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:20.384299 11231 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
W20260321 02:59:20.385599 13490 log.cc:927] Time spent T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260321 02:59:20.383378 13425 consensus_queue.cc:260] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: false } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.368393 13420 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:20.395218 13420 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:20.352972 13339 consensus_queue.cc:1048] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260321 02:59:20.414144 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:20.415126 11235 raft_consensus.cc:2468] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:20.417187 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:20.417984 11233 raft_consensus.cc:2468] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:20.421094 13425 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:20.422187 13134 raft_consensus.cc:2804] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:20.422746 13134 raft_consensus.cc:697] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:20.423647 13134 consensus_queue.cc:237] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.356608 13383 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:20.416404 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:20.432891 11231 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
W20260321 02:59:20.435199 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 63366.3us
I20260321 02:59:20.373083 13427 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
W20260321 02:59:20.451148 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 87405.3us
I20260321 02:59:20.457154 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:20.454174 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 136515us
I20260321 02:59:20.467242 11376 raft_consensus.cc:1240] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 2. Current term is 3. Ops: []
I20260321 02:59:20.458127 11233 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:20.436890 13383 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:20.473193 11349 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:20.491993 13401 raft_consensus.cc:2804] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:20.499886 11349 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:20.500955 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:20.517941 11275 leader_election.cc:304] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
W20260321 02:59:20.522398 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:20.526854 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:20.495805 13401 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:20.475919 13505 consensus_queue.cc:1059] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260321 02:59:20.541520 13139 consensus_queue.cc:1059] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:59:20.542301 13096 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:20.542665 13306 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.543421 13331 raft_consensus.cc:2749] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260321 02:59:20.572731 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 54897.6us
I20260321 02:59:20.544909 11972 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
W20260321 02:59:20.547381 13499 log.cc:927] Time spent T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.080s user 0.001s sys 0.000s
I20260321 02:59:20.520110 13445 raft_consensus.cc:2804] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:20.585503 13445 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:20.563680 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:20.586035 13445 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:20.556442 13084 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:20.552085 13401 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:20.595134 13508 raft_consensus.cc:3055] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Stepping down as leader of term 2
W20260321 02:59:20.605962 13503 log.cc:927] Time spent T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.128s user 0.000s sys 0.001s
W20260321 02:59:20.607019 13501 log.cc:927] Time spent T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260321 02:59:20.607451 13139 raft_consensus.cc:3055] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:20.608829 11975 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:20.540163 13511 raft_consensus.cc:2804] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:20.615015 13506 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:20.630012 11134 catalog_manager.cc:5671] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:59:20.624688 13508 raft_consensus.cc:740] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:20.634048 13306 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:20.635511 11276 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:20.573235 13096 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:20.637099 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 118910us
I20260321 02:59:20.642462 11972 raft_consensus.cc:1240] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:20.642931 13512 consensus_queue.cc:1059] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260321 02:59:20.643808 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 70027.1us
I20260321 02:59:20.646891 11132 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42514 (request call id 210) took 5349 ms. Trace:
W20260321 02:59:20.654085 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 181268us
I20260321 02:59:20.659192 11350 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:20.656042 13506 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:20.650379 13511 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:20.663595 11275 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:20.665199 11134 catalog_manager.cc:5671] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 1 to 3, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 3 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:59:20.650331 13459 raft_consensus.cc:2764] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260321 02:59:20.644006 13139 raft_consensus.cc:740] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:20.655205 13461 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:20.669902 11200 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:20.673969 11275 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:20.675185 11275 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:20.677074 11377 raft_consensus.cc:1275] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260321 02:59:20.658075 13235 raft_consensus.cc:2804] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:20.672116 11200 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:20.661068 13291 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.672673 13264 raft_consensus.cc:2804] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:20.670156 13424 raft_consensus.cc:2804] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:20.689162 11134 catalog_manager.cc:5671] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 3 to 5, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 5 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:20.669946 13511 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:20.663352 11132 rpcz_store.cc:276] 0321 02:59:15.297579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:15.297815 (+ 236us) service_pool.cc:225] Handling call
0321 02:59:20.646824 (+5349009us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:59:19.895068 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:59:19.895661 (+ 593us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:59:19.895717 (+ 56us) write_op.cc:435] Acquired schema lock
0321 02:59:19.895738 (+ 21us) tablet.cc:661] Decoding operations
0321 02:59:19.896382 (+ 644us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:59:19.896454 (+ 72us) write_op.cc:641] Partition lock acquired for write op
0321 02:59:19.896484 (+ 30us) tablet.cc:684] Acquiring locks for 3 operations
0321 02:59:19.896894 (+ 410us) tablet.cc:700] Row locks acquired
0321 02:59:19.896913 (+ 19us) write_op.cc:260] PREPARE: finished
0321 02:59:19.897067 (+ 154us) write_op.cc:270] Start()
0321 02:59:19.897201 (+ 134us) write_op.cc:276] Timestamp: P: 1774061959897004 usec, L: 0
0321 02:59:19.897221 (+ 20us) op_driver.cc:348] REPLICATION: starting
0321 02:59:19.911736 (+ 14515us) log.cc:844] Serialized 1299 byte log entry
0321 02:59:20.105687 (+193951us) op_driver.cc:464] REPLICATION: finished
0321 02:59:20.359374 (+253687us) write_op.cc:301] APPLY: starting
0321 02:59:20.361869 (+ 2495us) tablet.cc:1366] starting BulkCheckPresence
0321 02:59:20.362066 (+ 197us) tablet.cc:1369] finished BulkCheckPresence
0321 02:59:20.362231 (+ 165us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:59:20.384641 (+ 22410us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:59:20.384889 (+ 248us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0321 02:59:20.384959 (+ 70us) write_op.cc:312] APPLY: finished
0321 02:59:20.385331 (+ 372us) log.cc:844] Serialized 35 byte log entry
0321 02:59:20.385587 (+ 256us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:20.385721 (+ 134us) write_op.cc:454] Released schema lock
0321 02:59:20.385805 (+ 84us) write_op.cc:341] FINISH: Updating metrics
0321 02:59:20.386329 (+ 524us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:20.386374 (+ 45us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":30,"child_traces":[["op",{"apply.queue_time_us":244447,"apply.run_cpu_time_us":3495,"apply.run_wall_time_us":30036,"num_ops":3,"prepare.queue_time_us":162835,"prepare.run_cpu_time_us":3736,"prepare.run_wall_time_us":57330,"replication_time_us":196459,"thread_start_us":241023,"threads_started":2}]]}
I20260321 02:59:20.667623 13331 consensus_queue.cc:1048] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260321 02:59:20.712666 11275 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
W20260321 02:59:20.758261 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 86232.6us
I20260321 02:59:20.716001 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:20.759161 11377 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:20.759428 11200 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:20.680549 13412 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:20.715207 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:20.767477 11134 catalog_manager.cc:5671] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 2 to 3, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 3 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:20.696115 13264 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:20.771489 11200 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:20.699512 13339 raft_consensus.cc:3055] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:20.693899 13517 raft_consensus.cc:2804] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:20.773028 11200 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:20.775342 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.697713 13516 raft_consensus.cc:2804] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:20.714972 13424 raft_consensus.cc:697] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:20.777976 11200 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:20.766377 13412 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
W20260321 02:59:20.792176 13519 log.cc:927] Time spent T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
I20260321 02:59:20.774436 13084 raft_consensus.cc:2749] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:20.773981 13264 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:20.698046 13134 consensus_queue.cc:1059] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
I20260321 02:59:20.704537 13383 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:20.809134 13470 log.cc:927] Time spent T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260321 02:59:20.809249 13523 raft_consensus.cc:2804] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:20.810258 13331 raft_consensus.cc:2749] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:20.811215 13506 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:20.813509 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.777034 13339 raft_consensus.cc:740] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:20.775691 13517 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:20.778868 13516 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:20.823019 13291 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:20.823805 13096 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:20.813465 13076 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:20.835579 13076 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:20.836356 13424 consensus_queue.cc:237] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:20.836686 13203 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.837721 13383 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:20.839072 13203 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:20.838757 13339 consensus_queue.cc:260] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.839849 13339 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:20.842547 13445 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.832752 13516 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:20.793577 13412 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:20.757395 13139 consensus_queue.cc:260] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.847249 11378 raft_consensus.cc:1240] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260321 02:59:20.844111 11382 raft_consensus.cc:1240] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6. Ops: []
W20260321 02:59:20.849207 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 165488us
I20260321 02:59:20.851308 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:20.826889 13523 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:20.852952 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.866204 11385 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:20.866787 11377 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:20.855039 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.869297 11378 raft_consensus.cc:2393] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 7: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:20.818539 11134 catalog_manager.cc:5671] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: config changed from index -1 to 3, no longer has a pending config: VOTER 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1), VOTER 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2), VOTER fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 4 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } health_report { overall_health: HEALTHY } } }
I20260321 02:59:20.874014 13235 raft_consensus.cc:697] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:20.833189 13084 raft_consensus.cc:2804] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:20.832531 13506 raft_consensus.cc:2749] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:20.881537 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.832262 13517 raft_consensus.cc:3060] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:20.769498 13508 consensus_queue.cc:260] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:20.870342 13096 raft_consensus.cc:2804] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260321 02:59:20.920444 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 143182us
I20260321 02:59:20.921072 13235 consensus_queue.cc:237] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:20.849566 13427 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.922354 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.921710 11200 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:20.870160 13523 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:20.887738 13139 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:20.884914 13084 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:20.882340 13291 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:20.922210 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:20.931162 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:20.929754 11200 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:20.900832 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:20.953969 11972 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:20.954452 11382 raft_consensus.cc:1240] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:20.956207 13291 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.957398 11134 catalog_manager.cc:5671] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 3 to 5, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 5 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:20.958361 13291 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:20.959396 13420 consensus_meta.cc:287] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.564s user 0.012s sys 0.000s
I20260321 02:59:20.959858 13420 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.962854 13445 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:20.940744 13401 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:20.941776 13530 raft_consensus.cc:2804] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:20.963935 13530 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:20.964358 13530 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:20.944011 13506 raft_consensus.cc:2804] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:20.946231 11200 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:20.959028 11134 catalog_manager.cc:5671] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 3 to 6, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 6 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:20.919586 13096 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:20.972922 11942 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:20.967852 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:20.973793 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:20.974656 11309 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:20.976619 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:20.977035 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:20.978003 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.978355 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:20.978701 11381 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:20.979012 11302 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:20.979933 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.980263 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.980863 11384 raft_consensus.cc:2410] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260321 02:59:20.980933 11378 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:20.981645 11377 raft_consensus.cc:1275] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260321 02:59:20.981788 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.982254 11377 pending_rounds.cc:85] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5: Aborting all ops after (but not including) 2
I20260321 02:59:20.982635 11377 pending_rounds.cc:107] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260321 02:59:20.982553 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.982839 11385 raft_consensus.cc:2410] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260321 02:59:20.982991 11377 raft_consensus.cc:2889] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:20.984136 13076 raft_consensus.cc:3055] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:20.984542 13076 raft_consensus.cc:740] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:20.932809 13084 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:20.985677 11942 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:20.987290 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.994072 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:20.994985 11305 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:20.999099 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:21.000639 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:21.013643 11311 raft_consensus.cc:1275] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260321 02:59:21.014401 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:20.966441 13506 raft_consensus.cc:697] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:21.013312 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:21.009347 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:21.022751 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.966948 13096 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:21.024138 11308 raft_consensus.cc:2468] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:21.021857 11383 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:21.029682 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:21.023093 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:20.967909 13420 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:21.026186 11302 raft_consensus.cc:1275] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260321 02:59:21.026459 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:21.034077 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:21.036818 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:21.000206 13461 raft_consensus.cc:2804] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:21.040365 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:21.034188 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:21.044287 11307 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:21.013089 13322 raft_consensus.cc:2804] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:21.047348 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:20.973646 13291 raft_consensus.cc:2804] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:21.043413 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:21.001019 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:21.049886 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:21.054435 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:21.055133 11381 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:21.056897 13531 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:21.057482 13531 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:21.059693 13531 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:21.041185 13512 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:21.060726 13512 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.053771 13461 raft_consensus.cc:697] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:21.062734 13401 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:21.063063 13461 consensus_queue.cc:237] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:21.063617 11200 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:21.064095 13322 raft_consensus.cc:697] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:21.064715 13531 raft_consensus.cc:2749] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:21.065030 13322 consensus_queue.cc:237] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:21.068668 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:21.069380 13508 raft_consensus.cc:3060] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:21.069697 13203 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:21.070216 13203 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.072234 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:21.072357 13203 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:21.073664 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:21.074290 11378 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:21.074740 13264 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.075384 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:21.076084 11377 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
W20260321 02:59:21.076841 13520 log.cc:927] Time spent T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.267s user 0.002s sys 0.000s
I20260321 02:59:21.067056 11200 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:21.078998 13517 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.052865 13076 consensus_queue.cc:260] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:21.081987 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:21.082616 11381 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:21.082122 13076 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:21.084946 13203 raft_consensus.cc:2749] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:21.090692 11200 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:21.090920 11201 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:20.993368 13513 consensus_queue.cc:1048] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20260321 02:59:21.098217 13517 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:21.100348 13203 raft_consensus.cc:2804] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:21.100915 13203 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:21.101277 13203 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:21.101100 13412 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.101732 13264 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:21.103219 13412 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:21.104686 13531 raft_consensus.cc:2804] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:21.105106 13531 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:21.105441 13531 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:21.108330 13291 raft_consensus.cc:697] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:21.109449 13291 consensus_queue.cc:237] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:21.116676 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:21.120556 13506 consensus_queue.cc:237] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.124377 11975 raft_consensus.cc:1240] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:21.126910 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:21.127933 11380 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:21.140076 11376 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:21.140432 11383 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:21.142580 11200 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:21.144692 11976 raft_consensus.cc:1275] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260321 02:59:21.145476 11309 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:21.151533 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:21.151989 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:21.152318 11381 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:21.155185 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:21.159929 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:21.115280 13523 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.154384 13134 consensus_queue.cc:1048] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
W20260321 02:59:21.175557 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:21.147672 13420 raft_consensus.cc:2804] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election won for term 6
W20260321 02:59:21.184689 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:21.193351 11200 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:21.206894 11380 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:21.146225 13203 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.148540 13516 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.195464 13427 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:21.140775 13531 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:21.195690 13420 raft_consensus.cc:697] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:21.221024 13424 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:21.261996 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 68765.5us
W20260321 02:59:21.268888 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 46329us
I20260321 02:59:21.236335 13518 raft_consensus.cc:2804] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:21.271896 11200 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:21.249615 13512 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:21.261952 11972 raft_consensus.cc:1275] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260321 02:59:21.260093 13445 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:21.285614 13518 raft_consensus.cc:697] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:21.286846 11200 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:21.298645 11234 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0225614ab358492abd584946a97a42f2"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:21.299366 11234 raft_consensus.cc:606] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:21.300015 11234 raft_consensus.cc:614] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:21.281339 13537 raft_consensus.cc:2804] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:21.312415 13531 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:21.318645 11349 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:21.323896 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:21.324636 11975 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:21.268436 13424 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.289538 13401 consensus_queue.cc:1059] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260321 02:59:21.310261 13537 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:21.332301 13545 raft_consensus.cc:2804] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260321 02:59:21.357095 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 46042.5us
I20260321 02:59:21.358260 11349 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:21.359193 13545 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:21.340898 13339 raft_consensus.cc:2749] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:21.359829 13545 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
W20260321 02:59:21.361155 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 74528.9us
I20260321 02:59:21.295991 13445 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:21.354760 11942 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:21.367924 11234 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:21.368845 11234 raft_consensus.cc:606] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:21.369282 11234 raft_consensus.cc:614] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:21.372931 11200 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:21.355598 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:21.355865 13537 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:21.362111 13523 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:21.372447 13076 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:21.405484 13542 log.cc:927] Time spent T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.087s user 0.000s sys 0.001s
I20260321 02:59:21.405745 11975 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:21.407388 11201 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:21.350957 13291 consensus_queue.cc:1048] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260321 02:59:21.394894 13401 raft_consensus.cc:2749] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:21.410977 11234 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
W20260321 02:59:21.412143 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 54004.6us
I20260321 02:59:21.401060 13445 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:21.402137 13427 raft_consensus.cc:2804] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260321 02:59:21.417850 11275 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.660s user 0.015s sys 0.032s
I20260321 02:59:21.403270 13203 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:21.418175 11275 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.660s user 0.015s sys 0.032s
I20260321 02:59:21.362516 13420 consensus_queue.cc:237] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.417651 13548 consensus_queue.cc:1048] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
W20260321 02:59:21.420547 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 707977us
W20260321 02:59:21.428193 13535 log.cc:927] Time spent T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.134s user 0.001s sys 0.000s
I20260321 02:59:21.432869 11349 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:21.403170 13544 consensus_queue.cc:1048] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
W20260321 02:59:21.438447 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 50871.5us
I20260321 02:59:21.413065 13516 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:21.440316 11200 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:21.416424 13427 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:21.443500 11275 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:21.441809 11200 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:21.409691 13076 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.442204 11349 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:21.392867 13518 consensus_queue.cc:237] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.443094 13203 raft_consensus.cc:2804] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:21.442839 13549 raft_consensus.cc:2804] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:21.445762 13536 raft_consensus.cc:2804] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:21.445848 11275 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:21.455185 11234 raft_consensus.cc:606] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:21.443069 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:21.455634 11234 raft_consensus.cc:614] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:21.444669 13427 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:21.455482 11275 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:21.452829 13512 raft_consensus.cc:2749] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:21.478542 13533 raft_consensus.cc:2749] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:21.437745 13084 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
W20260321 02:59:21.460551 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:21.458978 13235 raft_consensus.cc:2804] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:21.464422 13339 raft_consensus.cc:3055] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Stepping down as leader of term 4
W20260321 02:59:21.490125 13546 log.cc:927] Time spent T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.050s user 0.000s sys 0.001s
I20260321 02:59:21.490377 13339 raft_consensus.cc:740] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:21.451788 13549 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:21.491456 13549 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:21.479789 13412 consensus_queue.cc:1048] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.012s
I20260321 02:59:21.492903 13096 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.495080 13096 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:21.462409 13551 consensus_queue.cc:1048] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260321 02:59:21.495913 11379 raft_consensus.cc:1275] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260321 02:59:21.455068 13511 consensus_meta.cc:287] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.691s user 0.010s sys 0.002s
I20260321 02:59:21.464543 13076 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:21.448846 13516 raft_consensus.cc:2804] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:21.503937 13516 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:21.504388 13516 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:21.474005 13514 raft_consensus.cc:2804] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:21.483208 11275 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:21.510360 13543 log.cc:927] Time spent T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.143s user 0.000s sys 0.001s
I20260321 02:59:21.510497 11275 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:21.448451 13523 consensus_queue.cc:1048] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260321 02:59:21.511852 11275 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:21.513327 13523 raft_consensus.cc:2804] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:21.513999 13523 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:21.513738 13533 raft_consensus.cc:2804] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:21.514465 13523 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:21.514668 13533 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:21.515173 13533 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:21.515419 13412 raft_consensus.cc:2749] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:21.497215 13511 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.489537 13424 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:21.517868 13511 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:21.489878 13235 raft_consensus.cc:697] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 4 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:21.460378 13203 raft_consensus.cc:697] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:21.519309 13235 consensus_queue.cc:237] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.502998 13459 consensus_queue.cc:1048] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260321 02:59:21.489632 13084 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:21.491304 13339 consensus_queue.cc:260] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.513367 11275 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:21.530174 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:21.543754 11275 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:21.521477 11349 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:21.550352 11275 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:21.459477 13536 raft_consensus.cc:697] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
W20260321 02:59:21.556254 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:21.519680 13203 consensus_queue.cc:237] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:21.510480 13550 log.cc:927] Time spent T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
I20260321 02:59:21.554770 13551 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:21.557293 13264 raft_consensus.cc:2804] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:21.556072 13139 raft_consensus.cc:2804] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:21.574081 11234 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
W20260321 02:59:21.574172 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 85519.9us
I20260321 02:59:21.578941 11234 raft_consensus.cc:606] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:21.579433 11234 raft_consensus.cc:614] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:21.558305 13514 raft_consensus.cc:697] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:21.561476 13339 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:21.579384 11349 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:21.576876 13264 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:21.600525 11349 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:21.578202 13139 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:21.580269 13424 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:21.606931 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:21.607092 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 165344us
I20260321 02:59:21.609376 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:21.609866 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:21.616182 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:21.617038 11378 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:21.623984 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:21.626632 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:21.628495 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:21.629181 11385 raft_consensus.cc:1275] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:21.628566 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:21.630398 11309 raft_consensus.cc:2410] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260321 02:59:21.631548 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 80221.1us
I20260321 02:59:21.632578 11275 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:21.633359 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:21.633795 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:21.636963 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:21.637307 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:21.638051 11385 raft_consensus.cc:2393] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 6: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:21.638103 11302 raft_consensus.cc:2393] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 6: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:21.637025 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:21.638794 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:21.640692 11303 raft_consensus.cc:2393] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:21.641465 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:21.642216 11304 raft_consensus.cc:2393] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 6: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:21.642678 13511 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:21.643193 13511 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.645615 13511 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:21.647851 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:21.648530 11377 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 6
I20260321 02:59:21.648864 13511 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:21.649883 13511 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.651928 13511 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:21.653970 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:21.656184 11230 raft_consensus.cc:1240] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:21.656527 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:21.656579 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:21.656136 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:21.657536 11233 raft_consensus.cc:2393] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 4: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:21.657656 11380 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 7
I20260321 02:59:21.658385 11235 raft_consensus.cc:1240] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20260321 02:59:21.658463 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:21.659718 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:21.660336 11232 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:21.661464 11200 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:21.662060 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:21.662907 11235 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:21.663196 13511 raft_consensus.cc:2749] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260321 02:59:21.665548 11227 raft_consensus.cc:1240] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:21.663704 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:21.668633 11200 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:21.670850 13511 raft_consensus.cc:2749] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:21.671425 11200 leader_election.cc:304] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:21.673882 11235 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:21.673977 13511 raft_consensus.cc:2749] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:21.675361 11200 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:21.606911 13264 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:21.603878 13401 raft_consensus.cc:2804] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:21.694080 13401 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:21.694559 13401 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:21.683503 11380 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:21.654984 11275 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:21.685952 11232 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 7.
I20260321 02:59:21.688119 11377 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:21.614951 13139 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
W20260321 02:59:21.692943 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 92548.6us
I20260321 02:59:21.708199 13445 consensus_queue.cc:1059] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260321 02:59:21.709244 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 54753.9us
I20260321 02:59:21.710259 11275 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:21.716310 11233 raft_consensus.cc:1240] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260321 02:59:21.717823 11235 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:21.720124 11200 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:21.723119 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:21.723841 11382 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:21.723256 11235 raft_consensus.cc:606] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:21.724702 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:21.725381 11377 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:21.629276 13512 raft_consensus.cc:2804] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:21.769606 11382 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:21.769654 11200 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:21.771687 11200 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:21.772723 13544 raft_consensus.cc:2749] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260321 02:59:21.774426 13544 raft_consensus.cc:2804] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260321 02:59:21.731566 13530 consensus_meta.cc:287] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.764s user 0.008s sys 0.005s
I20260321 02:59:21.775259 13544 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:21.791749 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260321 02:59:21.726253 13506 raft_consensus.cc:2749] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:21.622175 13424 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: false } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.794241 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:59:21.802124 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:21.798213 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:21.804339 11382 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:21.733975 13512 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:21.809520 11235 raft_consensus.cc:1240] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260321 02:59:21.811568 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:21.762732 13553 raft_consensus.cc:2804] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
W20260321 02:59:21.825121 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 114976us
I20260321 02:59:21.802156 13544 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:21.826104 11275 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:21.832728 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:21.833842 11377 raft_consensus.cc:2393] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 6: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:21.837441 11382 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:21.623980 13536 consensus_queue.cc:237] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.813612 13512 raft_consensus.cc:3060] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:21.788829 13545 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.807267 13560 raft_consensus.cc:2804] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:21.864712 13560 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:21.857542 13139 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.870397 11377 raft_consensus.cc:1275] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260321 02:59:21.842020 13553 raft_consensus.cc:697] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:21.800724 13530 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.763281 13514 consensus_queue.cc:237] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:21.889060 11235 raft_consensus.cc:1275] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260321 02:59:21.818928 13427 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:21.897943 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:21.927838 11303 raft_consensus.cc:1275] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260321 02:59:21.932514 11303 pending_rounds.cc:85] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da: Aborting all ops after (but not including) 3
I20260321 02:59:21.932849 11303 pending_rounds.cc:107] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260321 02:59:21.933125 11303 raft_consensus.cc:2889] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:21.942593 11200 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:21.872494 13572 raft_consensus.cc:2804] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:21.960779 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:21.964028 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:21.965046 11377 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
W20260321 02:59:21.967727 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:21.968199 13563 log.cc:927] Time spent T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c: Append to log took a long time: real 0.071s user 0.001s sys 0.000s
I20260321 02:59:21.968534 13506 raft_consensus.cc:2749] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:21.975687 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:21.976392 11232 raft_consensus.cc:1275] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260321 02:59:21.976240 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:21.977131 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:21.977607 11229 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:21.977840 11227 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:21.976636 11233 raft_consensus.cc:2410] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260321 02:59:21.983103 11235 raft_consensus.cc:1275] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260321 02:59:21.880802 13560 raft_consensus.cc:3060] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:21.985208 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:21.976505 11200 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:22.039480 11236 raft_consensus.cc:1240] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:21.890769 13567 raft_consensus.cc:2804] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260321 02:59:22.044308 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:22.123095 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:22.152648 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:22.153278 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 288605us
I20260321 02:59:22.160799 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:22.172402 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:21.986601 13572 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:21.959717 13517 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:22.232194 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 255734us
I20260321 02:59:22.236088 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:22.236315 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:22.243928 11227 raft_consensus.cc:1240] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:22.243944 11200 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:22.244554 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 372467us
W20260321 02:59:22.249447 11942 leader_election.cc:336] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.536s (SENT)
I20260321 02:59:22.250281 11942 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:22.051688 13567 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:22.152434 13139 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:22.247141 13548 raft_consensus.cc:2749] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:22.246206 13572 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
W20260321 02:59:22.275615 13541 log.cc:927] Time spent T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260321 02:59:22.219305 13554 consensus_queue.cc:1048] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.487s
I20260321 02:59:22.264341 13567 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:22.327380 11235 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
W20260321 02:59:22.316048 13571 log.cc:927] Time spent T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
W20260321 02:59:22.317031 13561 log.cc:927] Time spent T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.163s user 0.000s sys 0.000s
I20260321 02:59:22.265726 13577 raft_consensus.cc:2804] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:22.249377 13517 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:22.314320 13548 raft_consensus.cc:2749] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:22.338328 13577 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:22.338881 13577 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
W20260321 02:59:22.339305 13562 log.cc:927] Time spent T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
W20260321 02:59:22.340440 13575 log.cc:927] Time spent T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20260321 02:59:22.345800 13587 consensus_queue.cc:1059] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260321 02:59:22.349443 13513 raft_consensus.cc:3055] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:22.350018 13513 raft_consensus.cc:740] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:22.353958 13577 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:22.359633 13523 consensus_meta.cc:287] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.845s user 0.012s sys 0.000s
I20260321 02:59:22.361078 13577 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:22.388468 13557 log.cc:927] Time spent T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.563s user 0.001s sys 0.001s
I20260321 02:59:22.390801 13427 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:22.391049 13569 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:22.327174 13505 raft_consensus.cc:2749] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260321 02:59:22.309772 13424 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:22.468333 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 75205.2us
I20260321 02:59:22.476429 13582 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:22.477321 13582 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:22.351227 13513 consensus_queue.cc:260] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:22.382956 13545 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:22.486757 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:22.490106 11235 raft_consensus.cc:1240] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7. Ops: [6.2-6.2]
I20260321 02:59:22.490980 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:22.491720 11229 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:22.479691 13582 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:22.461678 13322 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:22.509500 11233 raft_consensus.cc:1240] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:22.513702 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:22.514518 11349 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.776s user 0.002s sys 0.009s
I20260321 02:59:22.516245 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:22.514688 11232 raft_consensus.cc:2393] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 5: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:22.517314 11235 raft_consensus.cc:2393] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 7: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
W20260321 02:59:22.521039 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 287794us
I20260321 02:59:22.521919 11200 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:22.524349 11230 raft_consensus.cc:1275] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260321 02:59:22.526693 11200 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:22.509502 11307 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.785s user 0.005s sys 0.005s
I20260321 02:59:22.469635 13518 consensus_queue.cc:1048] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
W20260321 02:59:22.509701 13549 consensus_meta.cc:287] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 1.018s user 0.010s sys 0.001s
W20260321 02:59:22.529269 11307 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.805s user 0.006s sys 0.005s
W20260321 02:59:22.516579 11349 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.778s user 0.002s sys 0.009s
I20260321 02:59:22.457460 13523 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:22.509833 13445 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.800s user 0.003s sys 0.008s
I20260321 02:59:22.289480 13553 consensus_queue.cc:237] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:22.519263 13533 consensus_meta.cc:287] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 1.003s user 0.009s sys 0.003s
I20260321 02:59:22.511816 13530 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:22.547936 11349 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:22.539582 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:22.549316 11230 raft_consensus.cc:1240] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:22.557789 11235 raft_consensus.cc:1240] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260321 02:59:22.558837 11229 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:22.560361 11200 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:22.513568 13577 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:22.467891 13569 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:22.569430 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 290423us
I20260321 02:59:22.570844 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:22.533135 13545 raft_consensus.cc:2804] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
W20260321 02:59:22.545631 13445 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.836s user 0.003s sys 0.009s
I20260321 02:59:22.513485 13322 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:22.589558 11200 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:22.552027 13424 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:22.556339 13556 log.cc:927] Time spent T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.817s user 0.000s sys 0.004s
W20260321 02:59:22.593039 13420 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.003s user 0.005s sys 0.023s
W20260321 02:59:22.611018 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:22.629249 13573 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:22.592079 13530 raft_consensus.cc:2804] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:22.633833 13595 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:22.693480 13595 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:22.696914 13595 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:22.706924 13568 log.cc:927] Time spent T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.237s user 0.000s sys 0.002s
W20260321 02:59:22.712414 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 122971us
I20260321 02:59:22.713434 13593 raft_consensus.cc:2749] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:22.713562 13530 raft_consensus.cc:697] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:22.628696 13459 consensus_queue.cc:1059] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.086s
I20260321 02:59:22.715186 13459 raft_consensus.cc:3055] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Stepping down as leader of term 2
I20260321 02:59:22.714888 13530 consensus_queue.cc:237] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:22.671525 11231 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58200 (request call id 1582) took 1013 ms. Trace:
I20260321 02:59:22.562420 13076 raft_consensus.cc:2764] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20260321 02:59:22.651911 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:22.713443 11200 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:22.736490 11200 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:22.725039 11231 rpcz_store.cc:276] 0321 02:59:21.657829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:21.659269 (+ 1440us) service_pool.cc:225] Handling call
0321 02:59:21.659586 (+ 317us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:21.660009 (+ 423us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:21.660037 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:21.660506 (+ 469us) log.cc:844] Serialized 26 byte log entry
0321 02:59:21.797793 (+137287us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:59:21.797838 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:21.797916 (+ 78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:22.671242 (+873326us) raft_consensus.cc:1685] finished
0321 02:59:22.671274 (+ 32us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:22.671450 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":104,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":80,"thread_start_us":136619,"threads_started":1,"wal-append.queue_time_us":919855}
I20260321 02:59:22.715718 13459 raft_consensus.cc:740] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:22.740839 11200 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:22.645879 13420 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.064s user 0.007s sys 0.028s
I20260321 02:59:22.576613 13506 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:22.576728 11973 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45078 (request call id 428) took 1081 ms. Trace:
W20260321 02:59:22.671484 13576 log.cc:927] Time spent T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.361s user 0.002s sys 0.000s
W20260321 02:59:22.631146 13401 consensus_meta.cc:287] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.936s user 0.007s sys 0.006s
I20260321 02:59:22.768342 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:22.769210 11944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:22.769515 13594 consensus_queue.cc:1048] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
W20260321 02:59:22.772122 13339 consensus_meta.cc:287] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 1.185s user 0.015s sys 0.000s
W20260321 02:59:22.748631 13516 consensus_meta.cc:287] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 1.244s user 0.013s sys 0.000s
I20260321 02:59:22.561210 13533 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:22.775101 13551 consensus_meta.cc:287] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 1.202s user 0.008s sys 0.004s
I20260321 02:59:22.776300 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:22.777793 11307 raft_consensus.cc:1275] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:22.646494 13573 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:22.778193 13544 consensus_meta.cc:287] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.928s user 0.010s sys 0.002s
I20260321 02:59:22.660151 13590 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:22.856206 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:22.778692 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:22.690935 13537 consensus_meta.cc:287] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 1.292s user 0.007s sys 0.003s
W20260321 02:59:22.781136 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:22.692198 13601 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:22.595145 13523 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:22.691232 13513 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:22.691486 13600 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:22.631000 13545 raft_consensus.cc:697] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:22.823740 11973 rpcz_store.cc:276] 0321 02:59:21.476452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:21.477065 (+ 613us) service_pool.cc:225] Handling call
0321 02:59:21.477551 (+ 486us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:21.478097 (+ 546us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:21.478145 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:21.478868 (+ 723us) log.cc:844] Serialized 26 byte log entry
0321 02:59:21.622870 (+144002us) raft_consensus.cc:1645] Marking committed up to 4
0321 02:59:21.628349 (+ 5479us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:21.692305 (+ 63956us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:22.535655 (+843350us) raft_consensus.cc:1685] finished
0321 02:59:22.537743 (+ 2088us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:22.551905 (+ 14162us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3601,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":138,"thread_start_us":127691,"threads_started":1,"wal-append.queue_time_us":257836}
I20260321 02:59:22.879782 13545 consensus_queue.cc:237] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:22.677675 13579 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:22.755224 13383 raft_consensus.cc:2749] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:22.749018 13427 raft_consensus.cc:2804] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:22.882917 13579 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:22.883529 13427 raft_consensus.cc:697] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:22.884505 13427 consensus_queue.cc:237] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:22.885190 13579 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:22.834362 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:22.886448 11379 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:22.891077 13322 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:22.580711 13549 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:22.892537 11972 raft_consensus.cc:1275] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260321 02:59:22.893530 13549 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:22.839246 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:22.766472 13518 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:22.581986 13577 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:22.847244 11311 raft_consensus.cc:2393] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 7: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:22.777822 13569 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:22.854123 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:22.655676 13517 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:22.739742 13424 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:22.844040 13560 consensus_meta.cc:287] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.606s user 0.010s sys 0.000s
I20260321 02:59:22.863976 11200 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:22.765007 13585 consensus_queue.cc:1048] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.093s
I20260321 02:59:22.826285 13506 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:22.840130 13401 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:22.843967 13516 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:22.894207 11382 raft_consensus.cc:1240] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:22.851855 13551 raft_consensus.cc:2749] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:22.935477 11379 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:22.996395 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:23.017109 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 372923us
I20260321 02:59:22.856665 13420 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:23.025095 11381 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
W20260321 02:59:23.025290 11275 connection.cc:232] RPC call timeout handler was delayed by -1.7141918400000122s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1537s for a response
W20260321 02:59:23.028424 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 164596us
I20260321 02:59:22.855639 13592 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
W20260321 02:59:23.032382 11275 connection.cc:232] RPC call timeout handler was delayed by -1.3802918399999271s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1908s for a response
I20260321 02:59:23.033143 11236 raft_consensus.cc:1240] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7. Ops: []
W20260321 02:59:23.034844 11275 connection.cc:232] RPC call timeout handler was delayed by -1.1903918399999611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21190000000000003s for a response
I20260321 02:59:22.861951 13084 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:23.039325 11385 raft_consensus.cc:1240] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:23.038919 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:22.949757 13512 consensus_meta.cc:287] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 1.065s user 0.008s sys 0.003s
W20260321 02:59:22.996958 13572 consensus_meta.cc:287] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.683s user 0.006s sys 0.007s
W20260321 02:59:23.032814 13264 consensus_meta.cc:287] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 1.316s user 0.007s sys 0.006s
I20260321 02:59:23.041087 11379 raft_consensus.cc:1240] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6. Ops: []
W20260321 02:59:22.956048 13514 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.024s sys 0.060s
I20260321 02:59:23.044641 11380 raft_consensus.cc:1240] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:23.046154 11234 raft_consensus.cc:2368] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5.
I20260321 02:59:23.045982 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:23.040063 11383 raft_consensus.cc:2468] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:23.013067 13587 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:23.030854 13518 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:23.053900 11942 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:23.031440 13445 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:23.057798 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:23.032622 13593 raft_consensus.cc:2749] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:23.036911 13606 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:23.042649 13592 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:23.063789 13518 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:23.038102 13420 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:23.036617 13549 consensus_queue.cc:1048] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260321 02:59:22.864857 13544 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:22.949050 13583 consensus_queue.cc:1048] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.109s
I20260321 02:59:23.040194 13536 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:22.874956 13601 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:22.857301 13459 consensus_queue.cc:260] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:22.871552 13590 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.055495 13084 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.067672 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:23.054495 13514 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.692s user 0.024s sys 0.073s
I20260321 02:59:23.064642 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:23.064378 11349 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:23.093382 11379 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:23.083453 11384 raft_consensus.cc:1275] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:23.048898 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:23.047521 13523 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:23.048436 13512 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:22.937028 13600 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.027743 13560 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:22.867760 13537 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.082274 13459 raft_consensus.cc:3060] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 2 FOLLOWER]: Advancing to term 3
I20260321 02:59:23.068099 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:23.065768 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:23.179035 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:23.096472 11275 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:23.100095 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 243471us
I20260321 02:59:23.082787 13533 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:23.123432 11942 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:23.098209 13549 raft_consensus.cc:2749] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260321 02:59:23.164054 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:23.060289 13587 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:23.081888 13601 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:23.074605 13544 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:23.112941 11349 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:23.205319 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:23.068058 13573 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:23.182437 11377 raft_consensus.cc:1240] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:23.055100 13264 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.078622 13518 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.067149 13322 raft_consensus.cc:2804] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:23.072036 13445 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:23.096794 13567 consensus_meta.cc:287] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.705s user 0.006s sys 0.005s
I20260321 02:59:23.084206 13590 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:23.083731 13516 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:23.185067 11942 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:23.086330 13084 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:23.076108 13424 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:23.185052 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:23.070030 13592 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:23.126947 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:23.233217 13518 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:23.234169 13533 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:23.234860 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:23.234716 13533 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:23.071921 13420 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:23.186847 11378 raft_consensus.cc:1275] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
W20260321 02:59:23.187753 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 95334.1us
I20260321 02:59:23.187753 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:23.236213 11378 pending_rounds.cc:85] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5: Aborting all ops after (but not including) 3
I20260321 02:59:23.236712 11378 pending_rounds.cc:107] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260321 02:59:23.237159 11378 raft_consensus.cc:2889] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260321 02:59:23.091164 13358 consensus_queue.cc:1175] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "fef7f8cc55b043039131bd9b2229f0a5" responder_term: 2 status { last_received { term: 2 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 2 index: 5 } }
I20260321 02:59:23.052767 13572 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:23.190135 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 146590us
I20260321 02:59:23.193133 13600 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:23.193948 13560 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:23.195129 13577 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:23.195379 13587 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:23.196581 11233 raft_consensus.cc:2393] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 6: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
W20260321 02:59:23.197028 13611 log.cc:927] Time spent T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
I20260321 02:59:23.254631 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:23.197305 13553 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:23.114156 13589 raft_consensus.cc:3055] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:23.262621 13589 raft_consensus.cc:740] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:23.263820 13589 consensus_queue.cc:260] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.264859 13401 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:23.265205 13589 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:23.268065 13076 consensus_queue.cc:1048] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:23.270570 13445 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:23.234201 13512 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:23.273272 13533 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:23.273763 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:23.275182 13572 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:23.251909 13596 consensus_queue.cc:1059] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20260321 02:59:23.066190 13597 consensus_queue.cc:1048] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.122s
I20260321 02:59:23.276660 13596 raft_consensus.cc:3055] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Stepping down as leader of term 4
W20260321 02:59:23.205585 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 105417us
I20260321 02:59:23.178925 11381 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:23.211422 11379 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:23.111482 13595 consensus_queue.cc:1059] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.079s
W20260321 02:59:23.280021 11234 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58200 (request call id 1574) took 1627 ms (client timeout 2109 ms). Trace:
W20260321 02:59:23.280478 11234 rpcz_store.cc:269] 0321 02:59:21.652817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:21.653523 (+ 706us) service_pool.cc:225] Handling call
0321 02:59:23.279801 (+1626278us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b5400264b54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:23.279944 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3192478976}
I20260321 02:59:23.280920 11385 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45634 (request call id 283) took 1034 ms. Trace:
I20260321 02:59:23.281350 11385 rpcz_store.cc:276] 0321 02:59:22.246449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:22.246683 (+ 234us) service_pool.cc:225] Handling call
0321 02:59:22.271096 (+ 24413us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:23.039768 (+768672us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:23.280696 (+240928us) spinlock_profiling.cc:243] Waited 768 ms on lock 0x7b540024b8d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:23.280863 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1766304000}
I20260321 02:59:23.124876 13523 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:23.196631 13537 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:23.224713 13322 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:23.286357 13322 raft_consensus.cc:3060] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:23.286655 11307 raft_consensus.cc:1275] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:23.288703 13264 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:23.289556 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:23.067651 13606 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.073180 13506 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:23.293572 11380 raft_consensus.cc:1275] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:23.294229 13583 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:23.294754 11380 raft_consensus.cc:1240] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:23.294704 13583 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:23.237741 11275 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:23.296598 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 58927.9us
I20260321 02:59:23.297454 13583 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:23.299324 11275 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:23.246524 11306 raft_consensus.cc:2393] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 7: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:23.211277 13601 raft_consensus.cc:2804] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:23.249197 11383 raft_consensus.cc:2376] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6. Re-sending same reply.
I20260321 02:59:23.300671 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:23.301149 11306 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56630 (request call id 1549) took 1668 ms (client timeout 2011 ms). Trace:
I20260321 02:59:23.257478 11309 raft_consensus.cc:2393] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 6: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
W20260321 02:59:23.301646 11306 rpcz_store.cc:269] 0321 02:59:21.632898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:21.633042 (+ 144us) service_pool.cc:225] Handling call
0321 02:59:23.300956 (+1667914us) spinlock_profiling.cc:243] Waited 1.61 s on lock 0x7b54001e00d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:23.301089 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3705885056}
I20260321 02:59:23.236204 13567 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.303063 13512 raft_consensus.cc:3055] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:23.303503 13512 raft_consensus.cc:740] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:23.261387 13553 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.307554 11302 raft_consensus.cc:2393] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 6: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:23.277083 13596 raft_consensus.cc:740] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:23.184989 13235 raft_consensus.cc:2749] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:23.076031 13536 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:23.283229 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:23.317499 11379 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:23.317616 11234 raft_consensus.cc:1240] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:23.315641 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:23.320402 11385 raft_consensus.cc:2393] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 3: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:23.285751 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:23.299494 13523 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:23.243163 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:23.278018 11349 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:23.312716 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:23.313068 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:23.324122 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:23.312474 13583 consensus_queue.cc:1059] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260321 02:59:23.315022 13533 raft_consensus.cc:476] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Not starting pre-election -- already a leader
I20260321 02:59:23.313088 13264 raft_consensus.cc:2749] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:23.304319 13512 consensus_queue.cc:260] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.330303 13536 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:23.330451 13512 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:23.331146 13076 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:23.331534 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:23.331688 13076 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.333700 13076 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:23.335331 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:23.322074 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 44202.5us
I20260321 02:59:23.336145 13264 consensus_queue.cc:1059] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:23.337072 13264 consensus_queue.cc:1243] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer fef7f8cc55b043039131bd9b2229f0a5 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260321 02:59:23.337461 11349 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:23.338574 13264 raft_consensus.cc:3055] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:23.338681 11349 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:23.339078 13264 raft_consensus.cc:740] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:23.339442 13445 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:23.340942 11275 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:23.342615 11304 raft_consensus.cc:2410] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260321 02:59:23.324590 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:23.348603 11385 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 5
I20260321 02:59:23.336150 11309 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:23.360072 11379 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:23.281910 13339 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:23.366350 11275 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:23.304311 13573 raft_consensus.cc:2749] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:23.371527 11275 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:23.372506 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:23.377279 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:23.379680 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 60103.5us
I20260321 02:59:23.377374 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:23.376892 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:23.378778 11382 raft_consensus.cc:2410] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260321 02:59:23.380602 11379 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:23.381283 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:23.308513 13595 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:23.382858 11306 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:23.314270 13601 raft_consensus.cc:697] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:23.386590 11385 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:23.384521 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:23.376931 13606 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:23.388237 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:23.340812 13600 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:23.364193 13545 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:23.409089 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:23.410574 11275 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:23.393162 11309 raft_consensus.cc:2410] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260321 02:59:23.413908 13518 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:23.411676 11380 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:23.414237 13600 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.414497 13518 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.371049 13551 consensus_queue.cc:1048] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260321 02:59:23.416465 13600 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:23.416657 13518 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:23.376839 13339 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.417098 11234 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0225614ab358492abd584946a97a42f2"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:23.314079 13572 consensus_queue.cc:1059] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20260321 02:59:23.417709 11234 raft_consensus.cc:606] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:23.378075 13590 raft_consensus.cc:2804] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:23.418118 11234 raft_consensus.cc:614] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:23.418254 13590 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:23.418668 13590 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:23.373126 13537 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:23.366009 13445 raft_consensus.cc:697] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 7 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:23.340204 13264 consensus_queue.cc:260] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.2, Last appended by leader: 2, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.377110 13513 raft_consensus.cc:3055] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:23.433796 11275 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:23.421177 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 47231.6us
I20260321 02:59:23.379426 13533 raft_consensus.cc:2749] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:23.372434 13506 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:23.377910 13549 raft_consensus.cc:2804] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:23.423381 11380 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:23.423609 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:23.417662 11383 raft_consensus.cc:2393] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 6: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:23.475224 11308 raft_consensus.cc:2393] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:23.427196 11306 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:23.481312 11233 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2d71c397418a46d085c9974230b7f964"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:23.482188 11233 raft_consensus.cc:606] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:23.428210 11379 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:23.432327 11311 raft_consensus.cc:3055] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:23.437006 11200 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:23.438405 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:23.487423 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 50531.8us
I20260321 02:59:23.487532 11309 raft_consensus.cc:2393] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 6: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:23.440531 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:23.488144 11200 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:23.444388 13572 raft_consensus.cc:2749] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:23.445528 13445 consensus_queue.cc:237] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.433187 13513 raft_consensus.cc:740] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:23.446592 13564 raft_consensus.cc:2749] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:23.447044 13616 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:23.449682 13264 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:23.382087 13424 consensus_queue.cc:1059] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260321 02:59:23.410691 13595 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.499079 13513 consensus_queue.cc:260] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:23.506657 13513 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:23.420285 13560 raft_consensus.cc:2749] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260321 02:59:23.423014 13612 log.cc:927] Time spent T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
I20260321 02:59:23.388011 13523 consensus_queue.cc:1048] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20260321 02:59:23.476263 13590 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.427820 13567 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:23.452184 13506 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:23.424279 13545 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.512614 13545 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:23.464226 13630 consensus_queue.cc:1059] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260321 02:59:23.513929 13401 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:23.514468 13401 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.514954 13630 raft_consensus.cc:3055] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:23.515367 13630 raft_consensus.cc:740] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:23.516618 13401 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:23.516350 13630 consensus_queue.cc:260] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:23.462211 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:23.518402 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:23.432699 13322 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:23.468694 13553 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:23.469516 13601 consensus_queue.cc:237] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.416698 13596 consensus_queue.cc:260] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:23.472458 13597 raft_consensus.cc:3055] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:23.485725 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:23.486459 11311 raft_consensus.cc:740] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:23.459048 13536 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:23.473569 13235 raft_consensus.cc:2749] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:23.462144 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:23.490360 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:23.526729 11304 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 5
I20260321 02:59:23.442611 11305 raft_consensus.cc:1275] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260321 02:59:23.489360 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:23.428376 13537 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.531898 13536 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.533031 13630 raft_consensus.cc:3060] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:23.533623 13537 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:23.533987 13536 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:23.523623 11311 consensus_queue.cc:260] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.535231 11311 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:23.535838 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:23.497105 13616 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:23.499466 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:23.500785 13595 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:23.499114 13424 raft_consensus.cc:3055] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:23.492969 13516 raft_consensus.cc:2804] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:23.492673 13573 raft_consensus.cc:493] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:23.510852 13590 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:23.512703 13506 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:23.474227 11349 leader_election.cc:400] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5.
I20260321 02:59:23.561827 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:23.532024 11975 raft_consensus.cc:2368] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4.
I20260321 02:59:23.520995 11200 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:23.521231 11233 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:23.523748 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:23.568617 11233 raft_consensus.cc:606] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:23.568720 11379 raft_consensus.cc:3060] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:23.569096 11233 raft_consensus.cc:614] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260321 02:59:23.569689 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 74476.5us
I20260321 02:59:23.493393 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:23.572465 11307 raft_consensus.cc:2368] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5.
I20260321 02:59:23.523042 13597 raft_consensus.cc:740] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:23.537091 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:23.574115 11306 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:23.520347 13322 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:23.547505 11309 raft_consensus.cc:1240] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:23.577740 11305 raft_consensus.cc:1240] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:23.491053 13618 raft_consensus.cc:2804] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:23.476536 13549 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:23.562309 11349 leader_election.cc:403] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260321 02:59:23.567039 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:23.607102 11379 raft_consensus.cc:2468] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:23.563143 13424 raft_consensus.cc:740] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:23.567179 13583 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:23.581512 11200 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:23.562626 13516 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:23.608778 11305 raft_consensus.cc:1240] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260321 02:59:23.562479 11303 raft_consensus.cc:1240] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 6. Current term is 7. Ops: []
W20260321 02:59:23.612097 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 138051us
I20260321 02:59:23.575482 13593 raft_consensus.cc:2804] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:23.617853 11307 raft_consensus.cc:1240] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:23.616822 11303 raft_consensus.cc:1240] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260321 02:59:23.619264 11942 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:23.609630 13618 raft_consensus.cc:697] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:23.621021 11307 raft_consensus.cc:1240] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:23.620970 13618 consensus_queue.cc:237] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.621744 13616 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:23.629238 11233 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:23.630172 11233 raft_consensus.cc:606] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:23.606958 13549 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:23.632792 11308 raft_consensus.cc:1240] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20260321 02:59:23.569495 13573 raft_consensus.cc:515] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:23.644605 11306 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:23.615778 13590 raft_consensus.cc:2749] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5.
I20260321 02:59:23.616760 13516 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:23.582218 13524 consensus_queue.cc:1048] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.075s
I20260321 02:59:23.733079 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:23.737416 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
W20260321 02:59:23.720417 11943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:23.724749 11233 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:23.754849 11233 raft_consensus.cc:606] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:23.653626 11311 raft_consensus.cc:1275] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260321 02:59:23.671825 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:23.755368 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:23.650254 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 68795.5us
I20260321 02:59:23.753407 11303 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 5.
W20260321 02:59:23.761415 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:23.762962 11200 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:23.714461 13620 log.cc:927] Time spent T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
W20260321 02:59:23.766407 11349 connection.cc:232] RPC call timeout handler was delayed by -1.028859698999895s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16690000000000002s for a response
I20260321 02:59:23.767516 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:23.769008 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:23.724731 13638 log.cc:927] Time spent T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260321 02:59:23.777794 11229 raft_consensus.cc:1240] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:23.654474 13322 raft_consensus.cc:2804] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader election won for term 4
W20260321 02:59:23.785643 11275 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:23.787245 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:23.787904 11230 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:23.788138 11231 raft_consensus.cc:1240] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:23.618034 13583 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:23.833693 13583 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:23.703840 13424 consensus_queue.cc:260] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.786965 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:23.654040 13593 raft_consensus.cc:697] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:23.836524 11236 raft_consensus.cc:2393] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 5: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:23.836782 13593 consensus_queue.cc:237] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.837889 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:23.839145 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:23.839630 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:23.840428 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:23.840649 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:23.841046 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:23.841616 11229 raft_consensus.cc:2468] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:23.841666 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:23.842352 11236 raft_consensus.cc:2393] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 6: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:23.842543 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:23.842588 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:23.843290 11227 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:23.844086 13339 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:23.848380 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:23.849109 11228 raft_consensus.cc:3060] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:23.789283 13616 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:23.821996 13506 consensus_queue.cc:1048] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:59:23.824180 13322 raft_consensus.cc:697] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:23.824981 11942 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:23.826659 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:23.854195 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:23.876837 11229 raft_consensus.cc:2393] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 6: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
W20260321 02:59:23.856169 11943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:23.789744 11232 raft_consensus.cc:1240] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4. Ops: []
W20260321 02:59:23.878212 11943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:23.859556 11230 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:23.792452 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:23.880280 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:23.682179 13554 raft_consensus.cc:2804] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:23.859927 11228 raft_consensus.cc:2410] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260321 02:59:23.863440 11227 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
W20260321 02:59:23.863687 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 58923.9us
I20260321 02:59:23.864018 11201 leader_election.cc:400] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Vote denied by peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285) with higher term. Message: Invalid argument: T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4.
I20260321 02:59:23.869205 11304 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:23.872479 13322 consensus_queue.cc:237] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:23.873572 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 48750.7us
I20260321 02:59:23.869926 13567 consensus_queue.cc:1059] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260321 02:59:23.875770 13339 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:23.790170 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260321 02:59:23.650556 13597 consensus_queue.cc:260] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.757326 13545 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:23.864295 11311 raft_consensus.cc:1275] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260321 02:59:23.878428 13616 raft_consensus.cc:697] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:23.893182 11309 raft_consensus.cc:1275] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260321 02:59:23.899883 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:23.926170 11381 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 5.
I20260321 02:59:23.888028 13505 raft_consensus.cc:2804] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:23.902688 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:23.892459 11201 leader_election.cc:403] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260321 02:59:23.928026 11942 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:23.924540 11311 pending_rounds.cc:85] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da: Aborting all ops after (but not including) 5
I20260321 02:59:23.931057 11311 pending_rounds.cc:107] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260321 02:59:23.933785 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:23.933992 11311 raft_consensus.cc:2889] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:23.934376 11228 raft_consensus.cc:2468] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 2.
W20260321 02:59:23.934711 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 80057.3us
I20260321 02:59:23.938470 11976 raft_consensus.cc:2468] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:23.931788 11275 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:23.949000 13572 raft_consensus.cc:2804] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:23.949156 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:23.950644 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:23.953177 11231 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 5.
I20260321 02:59:23.930001 13573 leader_election.cc:290] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:23.959123 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:23.933307 13554 raft_consensus.cc:697] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:23.930337 13505 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:23.937718 13235 raft_consensus.cc:3055] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:23.940637 13511 raft_consensus.cc:3060] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 2 FOLLOWER]: Advancing to term 4
I20260321 02:59:23.960879 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:23.937893 13358 raft_consensus.cc:2804] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:23.906903 13339 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:23.976290 11302 raft_consensus.cc:2410] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 7 because replica has last-logged OpId of term: 6 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260321 02:59:23.969681 13572 raft_consensus.cc:697] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:23.969383 13567 consensus_queue.cc:1059] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20260321 02:59:23.929639 13545 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:23.984506 11275 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:23.943928 13523 consensus_queue.cc:1059] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260321 02:59:23.944711 13560 consensus_queue.cc:1048] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260321 02:59:23.971225 13505 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:23.972390 13235 raft_consensus.cc:740] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:23.973881 13358 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:24.000620 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 233605us
I20260321 02:59:23.986565 11972 raft_consensus.cc:1275] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260321 02:59:23.953245 13635 consensus_queue.cc:1059] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
W20260321 02:59:24.006456 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:24.007124 11275 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:24.000945 13358 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:23.982023 13573 consensus_queue.cc:1059] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.022s
I20260321 02:59:23.985905 13533 consensus_queue.cc:1059] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.028s
I20260321 02:59:23.995007 13564 raft_consensus.cc:3055] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Stepping down as leader of term 5
W20260321 02:59:24.026116 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 60005.4us
I20260321 02:59:24.028546 13635 raft_consensus.cc:3055] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:24.029052 13581 raft_consensus.cc:2749] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:24.052165 13564 raft_consensus.cc:740] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:24.052610 11350 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:24.003806 13445 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:24.052978 13076 consensus_queue.cc:1048] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:59:24.053607 13445 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.054095 13595 consensus_queue.cc:1059] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:59:24.054956 13595 consensus_queue.cc:1059] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:59:24.031364 11275 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:24.035784 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:24.032869 13545 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:24.038565 13592 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:24.031922 13549 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:24.056128 13551 raft_consensus.cc:3055] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:24.056878 13551 raft_consensus.cc:740] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:24.057168 11383 raft_consensus.cc:1240] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:24.003647 13579 raft_consensus.cc:2749] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:24.016551 13401 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:24.046587 13593 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:24.046586 13264 raft_consensus.cc:3055] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:24.035791 13596 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:24.058876 13264 raft_consensus.cc:740] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:24.059141 13606 raft_consensus.cc:2804] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:24.055933 13595 raft_consensus.cc:3055] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:24.040650 13235 consensus_queue.cc:260] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:24.056653 13592 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.039484 13506 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:24.052184 13635 raft_consensus.cc:740] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:24.061193 11200 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:24.017148 13590 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:24.053318 13564 consensus_queue.cc:260] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.056658 11975 raft_consensus.cc:2410] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260321 02:59:24.067070 11275 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:24.030920 13554 consensus_queue.cc:237] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.033113 13572 consensus_queue.cc:237] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:23.982837 13616 consensus_queue.cc:237] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.094533 11200 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:24.128832 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 61861.7us
I20260321 02:59:24.057770 13551 consensus_queue.cc:260] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.084939 13595 raft_consensus.cc:740] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:24.086791 13445 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:24.088037 13579 raft_consensus.cc:2804] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:24.097553 13545 raft_consensus.cc:2749] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:24.146867 11200 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:24.153293 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:24.058640 13401 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.140410 11275 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:24.058881 13593 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.154615 11200 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:24.178947 11200 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:24.132476 13235 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
W20260321 02:59:24.182451 11943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:24.059747 13264 consensus_queue.cc:260] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.136495 13564 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
W20260321 02:59:24.190994 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 139277us
I20260321 02:59:24.134657 13383 raft_consensus.cc:2749] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:24.102026 13506 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.152318 13579 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:24.173395 13594 raft_consensus.cc:2749] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:24.125211 13533 consensus_queue.cc:1048] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.074s
W20260321 02:59:24.280366 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:24.172425 13084 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:24.147538 13523 raft_consensus.cc:2804] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:24.123363 13590 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.222462 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:24.145897 13339 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:24.218554 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:24.228401 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:24.229799 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:24.230986 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:24.233287 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:24.210906 13553 raft_consensus.cc:2749] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:24.236610 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:24.236858 13424 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:24.252879 13581 raft_consensus.cc:3055] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:24.257512 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:24.260414 13635 consensus_queue.cc:260] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.262907 13597 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:24.264091 13583 consensus_queue.cc:1059] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260321 02:59:24.265938 11378 raft_consensus.cc:1275] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260321 02:59:24.266609 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:59:24.266829 13513 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:24.267858 11130 catalog_manager.cc:5671] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 5 to 6, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 6 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
W20260321 02:59:24.322782 11943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:24.278100 13601 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:24.284421 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:24.197047 13076 raft_consensus.cc:2749] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:24.292296 11384 raft_consensus.cc:2393] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 6: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:24.292363 13084 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.294607 13536 raft_consensus.cc:2804] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election won for term 5
W20260321 02:59:24.297011 13642 log.cc:927] Time spent T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.209s user 0.000s sys 0.001s
I20260321 02:59:24.297809 13590 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:24.302450 13593 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:24.279621 13579 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:24.307559 13264 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:24.308116 13523 raft_consensus.cc:697] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:24.222589 13551 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:24.310304 13506 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:24.324611 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:24.325762 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:24.320852 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:24.194978 13445 consensus_queue.cc:1059] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.096s
W20260321 02:59:24.331799 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:24.138752 13595 consensus_queue.cc:260] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.335319 11381 raft_consensus.cc:2393] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 7: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:24.340698 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:24.341566 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:24.306459 13549 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:24.320315 13401 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:24.285135 13560 consensus_queue.cc:1059] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20260321 02:59:24.317044 13592 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:24.327353 13581 raft_consensus.cc:740] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
W20260321 02:59:24.393034 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:24.340153 13635 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:24.323676 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:24.350225 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
W20260321 02:59:24.430840 13641 log.cc:927] Time spent T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.518s user 0.005s sys 0.000s
I20260321 02:59:24.405916 11376 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:24.350317 13590 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:24.408085 11383 raft_consensus.cc:2393] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 7: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:24.315474 13513 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:24.345094 13536 raft_consensus.cc:697] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:24.334729 13601 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.410382 11130 catalog_manager.cc:5671] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 5 to 6, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4). New cstate: current_term: 6 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
W20260321 02:59:24.437847 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 297488us
W20260321 02:59:24.350904 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 172063us
I20260321 02:59:24.425503 13533 raft_consensus.cc:3055] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:24.639801 11134 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42502 (request call id 225) took 8546 ms. Trace:
W20260321 02:59:24.641042 11349 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:24.403551 13459 raft_consensus.cc:3055] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:24.645237 11381 raft_consensus.cc:1240] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260321 02:59:24.464397 13523 consensus_queue.cc:237] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:24.627873 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:24.661881 11130 catalog_manager.cc:5671] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 3 to 6, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 6 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:59:24.663229 11200 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:24.665697 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:24.626518 13606 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.011s sys 0.015s
I20260321 02:59:24.640287 13595 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:24.677928 11275 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:24.692601 11275 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:24.642158 13549 raft_consensus.cc:3055] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:24.640626 13076 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:24.662025 13662 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:24.691270 11384 raft_consensus.cc:1240] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7. Ops: [4.3-6.4]
I20260321 02:59:24.690368 11130 catalog_manager.cc:5671] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 1 to 4, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 4 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
W20260321 02:59:24.707780 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:24.724895 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:24.725593 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:24.727895 11130 catalog_manager.cc:5671] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 3 to 4, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 4 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:59:24.730225 11130 catalog_manager.cc:5671] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 3 to 4, leader changed from 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 4 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260321 02:59:24.665354 13516 consensus_meta.cc:287] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.900s user 0.007s sys 0.011s
I20260321 02:59:24.717785 11235 raft_consensus.cc:1240] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:24.666880 13656 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:24.667819 11134 rpcz_store.cc:276] 0321 02:59:16.125095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:16.147601 (+ 22506us) service_pool.cc:225] Handling call
0321 02:59:24.488636 (+8341035us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:59:21.308141 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:59:21.406921 (+ 98780us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:59:21.406972 (+ 51us) write_op.cc:435] Acquired schema lock
0321 02:59:21.407001 (+ 29us) tablet.cc:661] Decoding operations
0321 02:59:21.408176 (+ 1175us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:59:21.408301 (+ 125us) write_op.cc:641] Partition lock acquired for write op
0321 02:59:21.408344 (+ 43us) tablet.cc:684] Acquiring locks for 7 operations
0321 02:59:21.451922 (+ 43578us) tablet.cc:700] Row locks acquired
0321 02:59:21.453918 (+ 1996us) write_op.cc:260] PREPARE: finished
0321 02:59:21.484331 (+ 30413us) write_op.cc:270] Start()
0321 02:59:21.496571 (+ 12240us) write_op.cc:276] Timestamp: P: 1774061961484251 usec, L: 0
0321 02:59:21.496593 (+ 22us) op_driver.cc:348] REPLICATION: starting
0321 02:59:21.569362 (+ 72769us) log.cc:844] Serialized 2819 byte log entry
0321 02:59:22.585207 (+1015845us) op_driver.cc:464] REPLICATION: finished
0321 02:59:23.276018 (+690811us) write_op.cc:301] APPLY: starting
0321 02:59:23.276107 (+ 89us) tablet.cc:1366] starting BulkCheckPresence
0321 02:59:23.276317 (+ 210us) tablet.cc:1369] finished BulkCheckPresence
0321 02:59:23.276457 (+ 140us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:59:23.409315 (+132858us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:59:23.445749 (+ 36434us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0321 02:59:23.445799 (+ 50us) write_op.cc:312] APPLY: finished
0321 02:59:23.446051 (+ 252us) log.cc:844] Serialized 59 byte log entry
0321 02:59:23.546580 (+100529us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:23.566908 (+ 20328us) write_op.cc:454] Released schema lock
0321 02:59:23.573502 (+ 6594us) write_op.cc:341] FINISH: Updating metrics
0321 02:59:23.611416 (+ 37914us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:23.617087 (+ 5671us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":29201,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":667795,"apply.run_cpu_time_us":11825,"apply.run_wall_time_us":308682,"num_ops":7,"prepare.queue_time_us":241518,"prepare.run_cpu_time_us":13664,"prepare.run_wall_time_us":336592,"raft.queue_time_us":555213,"raft.run_cpu_time_us":29811,"raft.run_wall_time_us":594619,"replication_time_us":1054478,"thread_start_us":949523,"threads_started":5,"wal-append.queue_time_us":333287,"wal-append.run_cpu_time_us":58402,"wal-append.run_wall_time_us":2575291}]]}
W20260321 02:59:24.667524 13511 consensus_meta.cc:287] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.688s user 0.012s sys 0.006s
I20260321 02:59:24.666697 13533 raft_consensus.cc:740] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
W20260321 02:59:24.821975 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 99047.2us
I20260321 02:59:24.732581 11130 catalog_manager.cc:5671] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
W20260321 02:59:24.691027 13606 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.631s user 0.011s sys 0.016s
W20260321 02:59:24.822290 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 129770us
W20260321 02:59:24.712131 13358 consensus_meta.cc:287] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 0.652s user 0.011s sys 0.000s
I20260321 02:59:24.674686 13459 raft_consensus.cc:740] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:24.841800 11275 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:24.821369 13322 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.025s sys 0.049s
I20260321 02:59:24.833069 11130 catalog_manager.cc:5671] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 3 to 4, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4). New cstate: current_term: 4 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:59:24.847867 11235 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58214 (request call id 1245) took 1010 ms. Trace:
I20260321 02:59:24.801431 13600 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:24.852015 11235 rpcz_store.cc:276] 0321 02:59:23.837526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:23.841789 (+ 4263us) service_pool.cc:225] Handling call
0321 02:59:23.842073 (+ 284us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:24.717525 (+875452us) spinlock_profiling.cc:243] Waited 3.59 ms on lock 0x7b5400124ba0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac4c41a0 00007fa8ac54e5a0 00007fa8ac54f611 00007fa8ac54c16f 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c
0321 02:59:24.825133 (+107608us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:24.831560 (+ 6427us) spinlock_profiling.cc:243] Waited 811 ms on lock 0x7b54002f8ad4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:24.847800 (+ 16240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1872970368}
I20260321 02:59:24.666407 13590 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:24.843477 11943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:24.854861 11309 raft_consensus.cc:1275] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:24.855767 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:24.832034 13572 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.013s sys 0.029s
I20260321 02:59:24.714644 13569 raft_consensus.cc:2749] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:24.703861 13549 raft_consensus.cc:740] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:24.819720 13339 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:24.802997 13573 raft_consensus.cc:2749] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260321 02:59:24.863090 11378 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.016s sys 0.067s
I20260321 02:59:24.800976 13657 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:24.865507 11943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:24.847887 13606 raft_consensus.cc:697] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:24.801249 13659 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:24.857175 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:24.823472 13401 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:24.884905 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:24.903867 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:24.840727 13567 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.024s sys 0.027s
W20260321 02:59:24.906232 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 64511.3us
I20260321 02:59:24.908604 11275 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:24.890455 11381 raft_consensus.cc:1275] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260321 02:59:24.909830 11275 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
W20260321 02:59:24.892264 11378 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.555s user 0.016s sys 0.067s
I20260321 02:59:24.885074 13139 raft_consensus.cc:2749] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:24.734397 13076 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.895356 13339 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.802205 13601 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:24.897969 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:24.851398 13322 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.553s user 0.025s sys 0.049s
I20260321 02:59:24.900269 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:24.822691 13594 consensus_queue.cc:1048] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20260321 02:59:24.687602 13581 consensus_queue.cc:260] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.847316 13511 raft_consensus.cc:2749] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285) with higher term. Message: Invalid argument: T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4.
I20260321 02:59:24.734792 13616 raft_consensus.cc:2749] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260321 02:59:24.865983 13572 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.545s user 0.013s sys 0.029s
I20260321 02:59:24.922196 11307 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:24.906140 11302 raft_consensus.cc:2410] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260321 02:59:24.868985 13084 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:24.842772 13553 consensus_queue.cc:1048] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
W20260321 02:59:24.914047 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 269410us
I20260321 02:59:24.936553 11942 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:24.841986 13516 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.922415 13577 raft_consensus.cc:2764] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260321 02:59:24.803017 13662 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:24.906486 13567 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.604s user 0.024s sys 0.028s
I20260321 02:59:24.855649 13358 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:24.925316 11309 raft_consensus.cc:1275] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260321 02:59:24.995357 11309 pending_rounds.cc:85] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da: Aborting all ops after (but not including) 3
I20260321 02:59:24.995200 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:24.995589 11309 pending_rounds.cc:107] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260321 02:59:24.995837 11309 raft_consensus.cc:2889] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:24.925958 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:24.887262 13536 consensus_queue.cc:237] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.902704 13657 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.928853 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:24.869637 13445 consensus_queue.cc:1059] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260321 02:59:24.869982 13600 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:24.943783 11943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:24.907924 13659 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:24.925115 13573 raft_consensus.cc:2804] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:24.907704 13401 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:25.027227 11350 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 75006.8us
I20260321 02:59:24.928701 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:25.034197 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:25.001015 11376 pending_rounds.cc:85] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5: Aborting all ops after (but not including) 3
I20260321 02:59:25.000929 11378 raft_consensus.cc:2410] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260321 02:59:25.001519 11229 raft_consensus.cc:2468] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
W20260321 02:59:25.002902 13505 consensus_meta.cc:287] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 1.000s user 0.022s sys 0.009s
I20260321 02:59:25.001929 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:24.907327 13533 consensus_queue.cc:260] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:24.994676 13572 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:25.006434 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:24.915721 11275 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:25.010243 13600 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:25.010957 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:25.011811 11230 raft_consensus.cc:1240] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260321 02:59:25.012337 11383 raft_consensus.cc:2410] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260321 02:59:25.014183 13076 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:25.014817 13664 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:25.014920 13553 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:25.016263 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:25.017477 13358 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:25.018918 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:25.023026 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:25.025981 11309 raft_consensus.cc:1275] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:25.025790 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:24.840062 13656 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:24.992221 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:24.997506 13596 consensus_meta.cc:287] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 0.938s user 0.009s sys 0.009s
I20260321 02:59:24.914388 13459 consensus_queue.cc:260] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:24.927747 13654 consensus_queue.cc:1048] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.081s
W20260321 02:59:25.077011 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:25.116115 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 200718us
I20260321 02:59:25.116911 11275 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:25.078747 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:25.118023 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:25.086335 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:25.119463 11235 raft_consensus.cc:2393] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 6: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:25.088626 11380 raft_consensus.cc:1275] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:25.089438 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:25.122241 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:25.089759 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.093143 11385 raft_consensus.cc:2410] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260321 02:59:25.095300 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:25.101220 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.096796 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:25.011840 13660 log.cc:927] Time spent T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20260321 02:59:25.074918 11384 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:25.076074 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:25.135155 11231 raft_consensus.cc:2393] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 6: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:25.127686 11378 raft_consensus.cc:2393] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 6: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:25.078341 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:25.135200 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:25.137760 11232 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:25.130636 11227 raft_consensus.cc:1240] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260321 02:59:24.935310 13549 consensus_queue.cc:260] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:25.141752 13551 consensus_meta.cc:287] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.734s user 0.007s sys 0.004s
I20260321 02:59:25.141709 13549 raft_consensus.cc:3060] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:25.024353 13513 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:25.087491 13672 log.cc:927] Time spent T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
W20260321 02:59:25.082013 13235 consensus_meta.cc:287] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.870s user 0.010s sys 0.001s
I20260321 02:59:25.151813 11308 raft_consensus.cc:3055] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:25.092253 11376 pending_rounds.cc:107] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 5.4
W20260321 02:59:25.086901 13661 log.cc:927] Time spent T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.187s user 0.000s sys 0.001s
I20260321 02:59:25.152349 11308 raft_consensus.cc:740] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
W20260321 02:59:25.088690 13597 consensus_meta.cc:287] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.755s user 0.009s sys 0.002s
I20260321 02:59:25.146587 11384 raft_consensus.cc:2410] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260321 02:59:25.127290 13564 consensus_meta.cc:287] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.905s user 0.012s sys 0.000s
W20260321 02:59:25.107632 13424 consensus_meta.cc:287] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.780s user 0.008s sys 0.000s
I20260321 02:59:25.090281 13533 raft_consensus.cc:3060] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:25.153848 11229 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59202 (request call id 185) took 1195 ms (client timeout 1517 ms). Trace:
I20260321 02:59:25.149935 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:25.155144 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:25.131229 13264 consensus_meta.cc:287] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.783s user 0.007s sys 0.004s
W20260321 02:59:25.137125 13579 consensus_meta.cc:287] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.736s user 0.008s sys 0.002s
I20260321 02:59:25.115295 13339 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:25.093609 13662 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:25.168293 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 51440.2us
I20260321 02:59:25.201258 11275 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:25.169412 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:25.170660 11232 raft_consensus.cc:1240] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:25.091717 13657 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:25.105569 13583 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:25.180467 11231 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:25.180680 11380 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:25.177655 11376 raft_consensus.cc:2917] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Aborting config change with OpId 5.4: Aborted: Op aborted by new leader
W20260321 02:59:25.181562 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:25.101861 13656 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:25.164976 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:25.213558 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:25.213989 11227 raft_consensus.cc:2393] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 4: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
W20260321 02:59:25.181691 11229 rpcz_store.cc:269] 0321 02:59:23.958226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:23.958912 (+ 686us) service_pool.cc:225] Handling call
0321 02:59:25.153646 (+1194734us) spinlock_profiling.cc:243] Waited 975 ms on lock 0x7b540007eed4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:25.153773 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2241901952}
I20260321 02:59:25.156176 13569 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:25.183825 11309 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:25.184616 11377 raft_consensus.cc:2410] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260321 02:59:25.171450 13076 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:25.185144 11310 raft_consensus.cc:1275] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 5. (term mismatch)
I20260321 02:59:25.217983 11310 pending_rounds.cc:85] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da: Aborting all ops after (but not including) 4
I20260321 02:59:25.186980 11228 raft_consensus.cc:1275] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260321 02:59:25.112169 13671 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:25.173430 13595 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:25.074854 13606 consensus_queue.cc:237] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:25.187472 11307 raft_consensus.cc:1275] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:25.092150 13572 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:25.188269 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:25.229485 11230 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:25.153165 11308 consensus_queue.cc:260] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:25.231071 11308 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:25.225301 13569 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:25.232327 13516 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:25.232862 13569 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:25.215286 13590 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:25.132449 13577 raft_consensus.cc:2749] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:25.115296 13573 raft_consensus.cc:697] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:25.198029 11233 raft_consensus.cc:1275] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:25.136830 13139 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:25.204567 11303 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:25.153769 13600 raft_consensus.cc:3055] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:25.182940 11311 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:25.171871 13567 raft_consensus.cc:2804] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:25.079538 13505 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:25.216173 11376 raft_consensus.cc:2889] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:25.216188 11275 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:25.217331 13662 raft_consensus.cc:2749] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260321 02:59:25.218293 11310 pending_rounds.cc:107] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da: Aborting uncommitted NO_OP operation due to leader change: 5.5
I20260321 02:59:25.256857 11310 raft_consensus.cc:2889] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:25.226399 13445 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:25.222307 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:25.265204 11232 raft_consensus.cc:2393] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:25.225620 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:25.243117 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:25.243217 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:25.243201 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 306705us
I20260321 02:59:25.244382 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:25.094343 13664 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:25.246770 11231 raft_consensus.cc:2410] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260321 02:59:25.094580 13553 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:25.275795 13553 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:25.276892 13573 consensus_queue.cc:237] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:25.278370 13581 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
W20260321 02:59:25.279449 11942 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.9.213.1:36717 timed out after 1.517s (SENT) [suppressed 16 similar messages]
I20260321 02:59:25.280198 11975 raft_consensus.cc:1240] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
W20260321 02:59:25.280406 13668 log.cc:927] Time spent T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.128s user 0.000s sys 0.000s
I20260321 02:59:25.282608 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:25.285620 13505 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:25.288264 11235 raft_consensus.cc:1240] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:25.288447 13653 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:25.288982 13653 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:25.253137 11275 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:25.292249 13653 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:25.269484 11349 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:25.226553 13076 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:25.272586 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:25.199556 13579 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:25.218289 13583 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:25.294895 11382 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:25.292155 11275 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:25.309046 11275 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:25.312203 11275 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:25.246423 13595 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:25.317831 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:25.228631 13671 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:25.260164 13654 raft_consensus.cc:2749] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:25.247862 13600 raft_consensus.cc:740] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:25.221930 11383 raft_consensus.cc:1275] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260321 02:59:25.320417 11232 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:25.272089 13567 raft_consensus.cc:697] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:25.250864 13139 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:25.315410 11275 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:25.328531 11308 raft_consensus.cc:1275] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260321 02:59:25.335323 11376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:45622 (request call id 7) took 8266 ms. Trace:
I20260321 02:59:25.339874 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:25.344587 11275 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:25.341650 11308 pending_rounds.cc:85] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da: Aborting all ops after (but not including) 3
I20260321 02:59:25.346405 11308 pending_rounds.cc:107] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260321 02:59:25.346730 11308 raft_consensus.cc:2889] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:25.317267 13662 raft_consensus.cc:2749] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:25.303516 13553 consensus_queue.cc:1059] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260321 02:59:25.316955 13560 raft_consensus.cc:2749] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:25.273320 13445 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:25.357386 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:25.374240 11383 raft_consensus.cc:2468] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 2.
I20260321 02:59:25.343381 11376 rpcz_store.cc:276] 0321 02:59:17.053105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:17.055693 (+ 2588us) service_pool.cc:225] Handling call
0321 02:59:17.072533 (+ 16840us) log.cc:844] Serialized 431 byte log entry
0321 02:59:25.309259 (+8236726us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11482,"raft.queue_time_us":521017,"raft.run_cpu_time_us":2118,"raft.run_wall_time_us":196283,"thread_start_us":807390,"threads_started":3,"wal-append.queue_time_us":552155,"wal-append.run_cpu_time_us":5895,"wal-append.run_wall_time_us":1423332}
I20260321 02:59:25.358767 11275 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:25.360235 11377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40950 (request call id 1204) took 1122 ms. Trace:
I20260321 02:59:25.376777 11942 leader_election.cc:304] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:25.320622 13635 raft_consensus.cc:2749] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:25.322342 13401 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:25.320317 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:25.358091 13630 raft_consensus.cc:2804] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:25.368471 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:25.362905 13264 raft_consensus.cc:2749] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260321 02:59:25.376986 11377 rpcz_store.cc:276] 0321 02:59:24.238028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:24.324295 (+ 86267us) service_pool.cc:225] Handling call
0321 02:59:25.343186 (+1018891us) spinlock_profiling.cc:243] Waited 781 ms on lock 0x7b540009b354. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:25.360154 (+ 16968us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1796191616}
I20260321 02:59:25.405640 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:25.406015 11384 raft_consensus.cc:1275] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:25.406045 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:25.406332 11231 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 7.
I20260321 02:59:25.386970 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:25.323055 13653 consensus_queue.cc:1048] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260321 02:59:25.408872 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:25.409488 11231 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:25.389832 11275 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:25.362257 13654 raft_consensus.cc:2749] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:25.361106 13564 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:25.375612 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:25.431322 11383 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:25.361696 13659 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:25.423767 11943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:25.434940 11275 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:25.388228 13505 raft_consensus.cc:2804] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:25.437444 11308 raft_consensus.cc:1240] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:25.438620 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.439872 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:25.443813 11303 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:25.443842 11309 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:25.446151 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:25.446753 11309 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
W20260321 02:59:25.448235 11943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:25.450488 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:25.464931 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:25.392601 13656 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:25.394968 13662 raft_consensus.cc:2804] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:25.451295 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:25.488030 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.488338 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.491192 11307 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
W20260321 02:59:25.496340 11943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:25.432384 13536 raft_consensus.cc:2749] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260321 02:59:25.449760 13505 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:25.514678 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:25.520406 11309 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
W20260321 02:59:25.522655 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:25.523533 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.449613 13264 raft_consensus.cc:2749] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:25.531571 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.533178 11309 raft_consensus.cc:2410] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260321 02:59:25.469756 13662 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:25.527364 11383 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 6.
I20260321 02:59:25.433461 13560 consensus_queue.cc:1059] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.086s
I20260321 02:59:25.468921 13553 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:25.538779 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.539788 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.542826 11942 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:25.544457 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.505684 11303 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:25.433629 13564 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:25.548252 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:25.551378 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:25.552352 11230 raft_consensus.cc:2393] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 3: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:25.499039 13523 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:25.553730 11383 raft_consensus.cc:1275] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:25.497510 13618 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:25.502255 13572 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:25.530915 13505 raft_consensus.cc:3060] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:25.578814 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:25.582269 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:25.589532 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.531769 13339 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:25.577418 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:25.575373 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:25.592339 11303 raft_consensus.cc:3060] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:25.593808 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.541577 13662 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:25.597535 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:25.537227 13076 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:25.559875 11349 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:25.432557 13600 consensus_queue.cc:260] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:25.489653 13659 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:25.497845 13656 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:25.601619 11303 raft_consensus.cc:2468] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
W20260321 02:59:25.603902 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:25.560688 13322 raft_consensus.cc:2804] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:25.577204 13579 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:25.619412 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:25.608876 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.614454 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.620723 11308 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
W20260321 02:59:25.605120 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 45282.2us
I20260321 02:59:25.616189 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.622514 11309 raft_consensus.cc:2468] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:25.626672 11349 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:25.619529 13664 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:25.602574 13339 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:25.644416 11382 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40950 (request call id 1196) took 1424 ms. Trace:
I20260321 02:59:25.560375 13553 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:25.632975 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.540532 13567 consensus_queue.cc:237] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:25.630565 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:25.579983 13401 consensus_queue.cc:1048] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.054s
I20260321 02:59:25.622334 13657 raft_consensus.cc:2804] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:25.642560 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:25.645102 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.664705 11383 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:25.644659 13424 raft_consensus.cc:2749] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:25.644853 11382 rpcz_store.cc:276] 0321 02:59:24.220556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:24.220730 (+ 174us) service_pool.cc:225] Handling call
0321 02:59:25.644229 (+1423499us) spinlock_profiling.cc:243] Waited 856 ms on lock 0x7b5400112454. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:25.644355 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1967976704}
I20260321 02:59:25.592777 13523 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:25.633285 13516 raft_consensus.cc:3055] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:25.655669 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:25.626698 13139 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:25.666157 11182 auto_rebalancer.cc:791] Removing replace marker failed: Op aborted by new leader
I20260321 02:59:25.678464 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:25.679651 11182 auto_rebalancer.cc:799] Could not move replica: Incomplete: tablet 2f5e054aa3c74a3282b3340fd3b1ea21, TS 83b12b883a7f4129a87dadf5a78f08da -> TS 3d8d3176d6ec407ea2a965556c2e2c56 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:25.680087 11182 auto_rebalancer.cc:284] scheduled replica move failed to complete: Incomplete: tablet 2f5e054aa3c74a3282b3340fd3b1ea21, TS 83b12b883a7f4129a87dadf5a78f08da -> TS 3d8d3176d6ec407ea2a965556c2e2c56 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:25.673278 13595 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:25.680450 13553 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:25.681228 13564 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:25.683256 13339 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:25.630988 13322 raft_consensus.cc:697] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:25.685168 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:25.685570 13322 consensus_queue.cc:237] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:25.686399 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:25.657147 11349 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:25.615504 13659 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:25.632274 13671 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:25.681329 13693 log.cc:927] Time spent T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20260321 02:59:25.693504 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:25.693821 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.694500 11307 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 7.
I20260321 02:59:25.594538 13618 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:25.695484 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.698477 13569 raft_consensus.cc:2749] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:25.698796 11377 raft_consensus.cc:1275] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
W20260321 02:59:25.699069 13681 log.cc:927] Time spent T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
W20260321 02:59:25.700273 13684 log.cc:927] Time spent T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.235s user 0.000s sys 0.001s
I20260321 02:59:25.700412 13459 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:25.639621 13675 consensus_queue.cc:1048] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.042s
I20260321 02:59:25.702996 13523 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:25.694700 13683 log.cc:927] Time spent T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260321 02:59:25.666199 13554 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:25.704082 13677 log.cc:927] Time spent T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.215s user 0.001s sys 0.000s
I20260321 02:59:25.704391 13554 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:25.705482 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.706370 13554 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:25.706933 11304 raft_consensus.cc:1275] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 6 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:25.707352 13630 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:25.707823 13630 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:25.710389 13583 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:25.711166 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.711835 11304 raft_consensus.cc:2376] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5. Re-sending same reply.
I20260321 02:59:25.712967 11308 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 7.
I20260321 02:59:25.680011 11382 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:25.678927 11977 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:25.717214 11307 raft_consensus.cc:2410] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 8 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260321 02:59:25.677569 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:25.719748 11384 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 5.
I20260321 02:59:25.720420 11276 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:25.683315 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:25.722114 11306 raft_consensus.cc:2376] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5. Re-sending same reply.
I20260321 02:59:25.722781 11383 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:25.678905 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:25.735989 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 78895.2us
I20260321 02:59:25.737149 11349 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:25.664266 13657 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:25.678570 13516 raft_consensus.cc:740] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:25.738667 13657 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:25.743283 13401 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:25.692106 13549 consensus_meta.cc:287] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.549s user 0.009s sys 0.002s
I20260321 02:59:25.749366 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:25.750550 11306 raft_consensus.cc:2410] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 7 because replica has last-logged OpId of term: 6 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260321 02:59:25.750759 11942 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:25.753465 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:25.754200 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:25.748804 13703 consensus_queue.cc:1048] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.011s
I20260321 02:59:25.743475 13516 consensus_queue.cc:260] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:25.778168 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 64071.2us
W20260321 02:59:25.986675 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 249577us
I20260321 02:59:25.780153 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:25.987881 11349 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:25.988304 11309 raft_consensus.cc:2376] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5. Re-sending same reply.
I20260321 02:59:25.794130 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.787992 11385 raft_consensus.cc:2468] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:25.706388 13264 raft_consensus.cc:3055] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:25.819698 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.997490 11310 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 7.
W20260321 02:59:25.904443 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:25.915830 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:25.992389 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:25.999253 11304 raft_consensus.cc:2410] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260321 02:59:25.912351 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:25.983492 13656 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:25.817770 11232 consensus_queue.cc:237] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
W20260321 02:59:26.009966 11379 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40950 (request call id 1206) took 1771 ms (client timeout 1690 ms). Trace:
W20260321 02:59:26.010689 11379 rpcz_store.cc:269] 0321 02:59:24.238661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:24.325472 (+ 86811us) service_pool.cc:225] Handling call
0321 02:59:26.009411 (+1683939us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b5400373f54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:26.009855 (+ 444us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3637463424}
I20260321 02:59:26.011472 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:26.017747 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.993544 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:25.993383 11378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1582) took 1084 ms. Trace:
I20260321 02:59:26.019645 11308 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 7.
W20260321 02:59:25.928210 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 257674us
I20260321 02:59:26.022006 11200 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:25.985430 11943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:26.024287 11943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:26.024801 11943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:25.949601 13685 log.cc:927] Time spent T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.412s user 0.000s sys 0.004s
I20260321 02:59:26.028535 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:25.884186 13533 consensus_meta.cc:287] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.684s user 0.009s sys 0.003s
I20260321 02:59:25.883770 13583 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:25.748838 13551 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:25.936673 13593 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:25.778775 13618 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:25.743901 13401 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:26.019567 11378 rpcz_store.cc:276] 0321 02:59:24.908878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:25.145691 (+236813us) service_pool.cc:225] Handling call
0321 02:59:25.174437 (+ 28746us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:25.226293 (+ 51856us) raft_consensus.cc:1534] Early marking committed up to index 3
0321 02:59:25.516698 (+290405us) log.cc:844] Serialized 14 byte log entry
0321 02:59:25.993008 (+476310us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:25.993068 (+ 60us) raft_consensus.cc:1645] Marking committed up to 3
0321 02:59:25.993137 (+ 69us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:25.993213 (+ 76us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:25.993326 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":292853,"prepare.queue_time_us":370633,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":40,"raft.queue_time_us":67508,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":8332,"thread_start_us":638384,"threads_started":3,"wal-append.queue_time_us":159274}
I20260321 02:59:25.755566 13671 raft_consensus.cc:2804] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:26.087313 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:26.279554 11379 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 5.
I20260321 02:59:26.271910 11303 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:26.273310 11235 raft_consensus.cc:1240] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:26.281661 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:26.282861 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:26.143668 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:26.274518 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:26.128083 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:26.109014 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:26.276702 11231 raft_consensus.cc:1240] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:26.277630 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:26.285512 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:26.285938 11309 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 7.
I20260321 02:59:26.277644 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:26.286402 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:26.076712 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 88889.8us
I20260321 02:59:26.289022 11302 raft_consensus.cc:2468] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:25.820499 13659 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:26.291144 11349 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:25.996923 13264 raft_consensus.cc:740] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:26.008212 13595 raft_consensus.cc:2749] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:26.008206 13572 raft_consensus.cc:2764] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260321 02:59:26.292462 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 250938us
I20260321 02:59:26.039916 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:26.272476 13459 consensus_meta.cc:287] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 0.571s user 0.010s sys 0.001s
I20260321 02:59:26.039325 13705 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:26.286633 11311 raft_consensus.cc:2376] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5. Re-sending same reply.
I20260321 02:59:26.132091 13564 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:26.299610 11231 raft_consensus.cc:1240] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:26.249208 13601 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:26.306202 11306 raft_consensus.cc:2368] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5.
I20260321 02:59:26.306149 11200 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:26.225677 13549 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:26.111023 13656 raft_consensus.cc:2804] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:26.087153 13424 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:26.316478 13705 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:26.308164 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:26.281044 13551 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:26.106520 13554 consensus_queue.cc:1048] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.222s
W20260321 02:59:26.309461 13505 consensus_meta.cc:287] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 0.698s user 0.010s sys 0.000s
I20260321 02:59:26.107362 13594 consensus_queue.cc:1048] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.202s
I20260321 02:59:26.279106 13671 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:26.310096 11973 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45098 (request call id 356) took 1412 ms. Trace:
W20260321 02:59:26.311022 13630 consensus_meta.cc:287] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 0.603s user 0.005s sys 0.004s
I20260321 02:59:26.290918 13597 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:26.132349 13523 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:26.124759 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:26.335294 13601 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:26.144307 13445 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:26.339146 11236 raft_consensus.cc:1240] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:26.371920 13523 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:26.373948 13601 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:26.361876 13618 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:26.374982 13523 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:26.372447 13702 log.cc:927] Time spent T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.103s user 0.000s sys 0.001s
I20260321 02:59:26.376066 13597 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:26.378031 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:26.075690 13699 consensus_queue.cc:1048] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.253s
I20260321 02:59:26.295411 13606 raft_consensus.cc:2749] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260321 02:59:26.349972 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 272151us
I20260321 02:59:26.317409 11200 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:26.292222 13593 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:26.287606 13139 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:26.354184 13705 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:26.334892 13564 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:26.384686 13139 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:26.386677 13564 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:26.334631 13358 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:26.374714 13618 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:26.287533 13583 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:26.375366 13630 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:26.387498 13358 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:26.360718 13551 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:26.336010 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:26.395100 11228 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:26.393761 13564 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:26.369447 13671 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Advancing to term 6
W20260321 02:59:26.406109 11275 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:26.349812 13656 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:26.386678 13139 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:26.420589 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:26.433432 11943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:26.438354 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:26.439086 11306 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 5.
I20260321 02:59:26.364099 13424 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:26.368667 13505 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:26.444325 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:26.448302 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 94265.7us
I20260321 02:59:26.451921 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:26.457258 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:26.458112 11303 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 5.
I20260321 02:59:26.452364 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:26.459852 11309 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 5.
I20260321 02:59:26.459139 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:26.345331 13549 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:26.462328 11349 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:26.466712 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:26.453564 11275 leader_election.cc:304] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:26.415337 13533 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:26.398069 13712 raft_consensus.cc:2804] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:26.463095 11303 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 5.
I20260321 02:59:26.422536 13656 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:26.468422 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:26.482477 11311 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:26.402128 13564 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:26.489156 11976 raft_consensus.cc:2368] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 3. Current term is 4.
I20260321 02:59:26.489553 11349 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:26.495931 13076 raft_consensus.cc:2804] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:26.497164 13076 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:26.497547 13076 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:26.506889 11349 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:26.506911 11275 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:26.509270 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 191963us
I20260321 02:59:26.481994 13712 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:26.483304 13618 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:26.422267 13264 consensus_queue.cc:260] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:26.529369 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:26.530491 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:26.530905 11234 raft_consensus.cc:1275] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260321 02:59:26.485131 13139 raft_consensus.cc:2749] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:26.466190 13523 consensus_queue.cc:1059] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260321 02:59:26.536377 13523 raft_consensus.cc:3055] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:26.536754 13523 raft_consensus.cc:740] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:26.536975 13567 consensus_queue.cc:1059] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:59:26.537598 13523 consensus_queue.cc:260] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:26.538626 13657 consensus_meta.cc:287] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.795s user 0.009s sys 0.000s
I20260321 02:59:26.538882 13424 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:26.539114 13657 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:26.526854 13717 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:26.367818 11973 rpcz_store.cc:276] 0321 02:59:24.897809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:24.900671 (+ 2862us) service_pool.cc:225] Handling call
0321 02:59:24.923762 (+ 23091us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:24.942316 (+ 18554us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:24.950150 (+ 7834us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:25.131538 (+181388us) log.cc:844] Serialized 26 byte log entry
0321 02:59:25.329888 (+198350us) raft_consensus.cc:1645] Marking committed up to 5
0321 02:59:25.336126 (+ 6238us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:25.342639 (+ 6513us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:25.815786 (+473147us) spinlock_profiling.cc:243] Waited 9.37 ms on lock 0x7b440002304c. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93c2dc5 00007fa8a93c37e2 00007fa8ac0f757b 00007fa8ac0f2e4f 00007fa8ac0e6c0e 00007fa8ac0e5f56 00007fa8ac0e5b78 00007fa8ac0ef5c1 00007fa8ac0ef579 00007fa8ac0ef509
0321 02:59:26.309774 (+493988us) raft_consensus.cc:1685] finished
0321 02:59:26.309802 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:26.310028 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":65908,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":8071,"spinlock_wait_cycles":21546368,"thread_start_us":121924,"threads_started":1,"wal-append.queue_time_us":267820}
I20260321 02:59:26.540135 13717 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:26.512547 13723 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:26.541471 13723 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:26.542460 13717 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:26.543581 13723 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:26.535595 13708 log.cc:927] Time spent T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260321 02:59:26.539490 13424 raft_consensus.cc:3060] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:26.534603 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:26.538900 13707 log.cc:927] Time spent T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20260321 02:59:26.488128 13593 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:26.495867 13654 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:26.547396 11311 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 6.
I20260321 02:59:26.531672 11229 raft_consensus.cc:1275] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:26.552358 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:26.549175 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:26.551168 11236 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:26.554311 13076 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:26.556203 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:26.487174 13459 consensus_queue.cc:1048] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260321 02:59:26.558869 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:26.561458 11228 raft_consensus.cc:2468] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:26.564953 11230 raft_consensus.cc:1240] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260321 02:59:26.565201 13564 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:26.565886 11200 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:26.566161 13714 log.cc:927] Time spent T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c: Append to log took a long time: real 0.086s user 0.000s sys 0.001s
I20260321 02:59:26.564874 13657 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:26.484189 13533 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:26.569643 11232 raft_consensus.cc:1275] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:26.512634 13659 raft_consensus.cc:2804] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:26.546061 13560 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:26.567392 11200 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:26.538926 13523 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:26.571560 13076 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:26.527176 13712 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:26.556217 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:26.572966 13560 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:26.584515 13560 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:26.585158 11977 raft_consensus.cc:1240] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:26.590086 13505 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:26.586633 13671 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:26.587858 13322 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:26.523116 13709 consensus_queue.cc:1048] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.074s
I20260321 02:59:26.588565 13699 raft_consensus.cc:2804] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:26.589148 13583 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:26.597211 13076 raft_consensus.cc:2804] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:26.572589 13659 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:26.572203 13401 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:26.602893 13076 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:26.597628 11200 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:26.603487 13076 raft_consensus.cc:3060] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:26.604492 11236 raft_consensus.cc:2410] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260321 02:59:26.601424 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 116374us
I20260321 02:59:26.605322 11942 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:26.605675 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 98869.8us
I20260321 02:59:26.606422 11349 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:26.607224 13657 raft_consensus.cc:2749] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:26.594411 13322 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:26.609848 11349 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
W20260321 02:59:26.619383 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 85922.9us
I20260321 02:59:26.568585 13654 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:26.619941 11305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1613) took 1051 ms. Trace:
I20260321 02:59:26.622367 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:26.625130 11303 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:26.625420 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:26.620221 11349 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:26.626154 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:26.602532 13699 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:26.628180 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:26.602880 13630 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:26.610615 13659 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:26.573912 13723 consensus_queue.cc:1048] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.031s
I20260321 02:59:26.634969 11236 raft_consensus.cc:1275] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260321 02:59:26.626780 11309 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:26.620994 13076 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:26.623329 11305 rpcz_store.cc:276] 0321 02:59:25.568720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:25.705416 (+136696us) service_pool.cc:225] Handling call
0321 02:59:25.705737 (+ 321us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:25.716314 (+ 10577us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:25.716344 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:25.740112 (+ 23768us) log.cc:844] Serialized 26 byte log entry
0321 02:59:26.126499 (+386387us) raft_consensus.cc:1645] Marking committed up to 4
0321 02:59:26.417191 (+290692us) log.cc:844] Serialized 14 byte log entry
0321 02:59:26.417476 (+ 285us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:26.417594 (+ 118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:26.619704 (+202110us) raft_consensus.cc:1685] finished
0321 02:59:26.619724 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:26.619875 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":279215,"prepare.queue_time_us":352573,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":28367,"raft.run_cpu_time_us":309,"raft.run_wall_time_us":26750,"thread_start_us":491075,"threads_started":2,"wal-append.queue_time_us":607574}
I20260321 02:59:26.619051 13583 raft_consensus.cc:2804] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:26.621678 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:26.637564 11311 raft_consensus.cc:2410] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260321 02:59:26.629302 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:26.633025 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:26.593102 13564 consensus_queue.cc:1059] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.032s
I20260321 02:59:26.625100 13549 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:26.650915 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 44800.2us
I20260321 02:59:26.639783 11349 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:26.622354 13401 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:26.665984 13076 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:26.666687 11303 raft_consensus.cc:2468] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:26.618875 13721 raft_consensus.cc:2804] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:26.629879 13724 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:26.678556 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:26.631549 13699 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:26.638492 13583 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:26.637684 13723 consensus_queue.cc:1243] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer 83b12b883a7f4129a87dadf5a78f08da log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260321 02:59:26.716986 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:26.717989 13654 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:26.704013 13721 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:26.718782 13721 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
W20260321 02:59:26.713366 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 73649.5us
I20260321 02:59:26.708994 13401 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:26.629592 13718 consensus_queue.cc:1048] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
I20260321 02:59:26.715198 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:26.636700 13657 raft_consensus.cc:2749] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:26.732347 11349 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:26.734076 11977 raft_consensus.cc:1275] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260321 02:59:26.734625 11349 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:26.736116 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:26.638020 13590 consensus_queue.cc:1048] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.083s
I20260321 02:59:26.748621 11378 raft_consensus.cc:1275] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260321 02:59:26.715337 13549 raft_consensus.cc:2749] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:26.716240 13583 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:26.640121 13705 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:26.753293 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:26.730185 13401 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:26.754047 11376 raft_consensus.cc:3060] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 2 FOLLOWER]: Advancing to term 4
I20260321 02:59:26.742162 13727 raft_consensus.cc:2749] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:26.710011 13724 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:26.772768 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 137370us
I20260321 02:59:26.773000 11942 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:26.775588 11200 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:26.751333 13590 consensus_queue.cc:1243] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer 83b12b883a7f4129a87dadf5a78f08da log is divergent from this leader: its last log entry 6.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260321 02:59:26.779892 11200 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:26.781913 11200 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:26.767107 13549 raft_consensus.cc:2804] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:26.784417 11200 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:26.794284 11376 raft_consensus.cc:2468] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:26.795689 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:26.796473 11378 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:26.797586 13076 raft_consensus.cc:2749] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:26.799137 11200 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:26.799624 11942 leader_election.cc:304] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:26.800483 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 65936us
I20260321 02:59:26.802594 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:26.806480 11200 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:26.810573 11200 leader_election.cc:304] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:26.811035 13733 consensus_queue.cc:1048] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:59:26.812976 13733 raft_consensus.cc:2749] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:26.813721 13733 raft_consensus.cc:2749] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:26.783488 13718 raft_consensus.cc:2749] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:26.822225 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:26.822991 11310 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:26.829164 13705 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:26.829766 13597 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:26.783375 13719 consensus_queue.cc:1048] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.047s
W20260321 02:59:26.831050 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:26.783432 13727 consensus_queue.cc:1048] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
W20260321 02:59:26.813730 13725 log.cc:927] Time spent T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.149s user 0.001s sys 0.000s
I20260321 02:59:26.779114 13723 raft_consensus.cc:2749] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:26.783349 13731 raft_consensus.cc:2804] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:26.807758 11349 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
W20260321 02:59:26.863030 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 55388.7us
I20260321 02:59:26.788221 13671 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:26.823488 11310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56616 (request call id 1659) took 1111 ms. Trace:
I20260321 02:59:26.799382 13630 consensus_queue.cc:1048] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260321 02:59:26.839087 13724 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:26.841138 13656 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:26.849747 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:26.850375 13662 consensus_meta.cc:287] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 1.231s user 0.009s sys 0.003s
I20260321 02:59:26.861021 11308 raft_consensus.cc:1275] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260321 02:59:26.872354 11308 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56616 (request call id 1660) took 1160 ms. Trace:
I20260321 02:59:26.864014 11349 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:26.874807 11349 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:26.877272 11349 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:26.872740 11308 rpcz_store.cc:276] 0321 02:59:25.711865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:26.074390 (+362525us) service_pool.cc:225] Handling call
0321 02:59:26.843701 (+769311us) spinlock_profiling.cc:243] Waited 3.61 ms on lock 0x7b5800002a98. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b1522680 00007fa8b150cfe9 00007fa8b150ca92 00007fa8aaa8ad34
0321 02:59:26.843912 (+ 211us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:26.872088 (+ 28176us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:26.872298 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8304512}
W20260321 02:59:26.884860 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:26.863162 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 63648.7us
I20260321 02:59:26.789615 13322 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:26.789428 13533 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:26.783268 13618 raft_consensus.cc:2749] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:26.865181 13669 raft_consensus.cc:2804] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader election won for term 4
I20260321 02:59:26.866375 11310 rpcz_store.cc:276] 0321 02:59:25.711580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:26.015151 (+303571us) service_pool.cc:225] Handling call
0321 02:59:26.822051 (+806900us) spinlock_profiling.cc:243] Waited 3.81 ms on lock 0x7b5800002a98. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b150e760 00007fa8b150e044 00007fa8aaa8f834 00007fa8aaa8f7c2
0321 02:59:26.823427 (+ 1376us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8752384}
I20260321 02:59:26.800402 13549 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:26.904217 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:26.865383 13731 raft_consensus.cc:697] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:26.908628 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:26.910358 11349 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:26.915247 11349 leader_election.cc:400] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267) with higher term. Message: Invalid argument: T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5.
I20260321 02:59:26.870118 13601 raft_consensus.cc:2804] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:26.916875 11349 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:26.886466 11942 leader_election.cc:336] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Timed out: RequestConsensusVote RPC to 127.9.213.1:36717 timed out after 1.517s (SENT)
I20260321 02:59:26.889479 13675 raft_consensus.cc:2749] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:26.888401 13516 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:26.889757 13727 raft_consensus.cc:2749] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:26.909937 13549 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:26.930047 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:26.930648 11384 raft_consensus.cc:1240] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260321 02:59:26.929461 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:26.930802 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 291838us
I20260321 02:59:26.931919 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:26.932555 11382 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:26.879715 13662 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:26.917177 13669 raft_consensus.cc:697] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:26.934845 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:26.937697 11382 raft_consensus.cc:2393] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 5: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
W20260321 02:59:26.938390 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 127936us
I20260321 02:59:26.939066 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:26.939237 11384 raft_consensus.cc:1275] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:26.941428 11376 raft_consensus.cc:1240] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 7. Current term is 8. Ops: [4.2-7.4]
I20260321 02:59:26.920801 13601 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:26.920475 13735 raft_consensus.cc:2804] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:26.945884 11275 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:26.928856 13590 raft_consensus.cc:2804] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:26.951721 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:26.927037 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:26.956396 11381 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:26.954924 11275 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:26.958948 11306 raft_consensus.cc:2393] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 8: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
W20260321 02:59:26.966769 13551 consensus_meta.cc:287] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.565s user 0.008s sys 0.001s
I20260321 02:59:26.961462 13654 raft_consensus.cc:2749] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:26.937354 13533 consensus_queue.cc:1048] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260321 02:59:26.955904 13590 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:26.970088 11976 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45088 (request call id 275) took 1602 ms (client timeout 1500 ms). Trace:
I20260321 02:59:27.005227 13590 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
W20260321 02:59:27.005404 11976 rpcz_store.cc:269] 0321 02:59:25.367919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:25.368156 (+ 237us) service_pool.cc:225] Handling call
0321 02:59:26.969836 (+1601680us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400474354. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:26.969975 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2518682880}
I20260321 02:59:27.006413 13659 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:26.972873 11376 raft_consensus.cc:1275] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260321 02:59:27.007301 11376 pending_rounds.cc:85] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5: Aborting all ops after (but not including) 2
I20260321 02:59:27.007562 11376 pending_rounds.cc:107] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260321 02:59:27.007838 11376 raft_consensus.cc:2889] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:27.008450 13656 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:26.962019 13705 consensus_queue.cc:1048] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20260321 02:59:27.012042 13597 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:27.012583 13597 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:27.013212 13424 raft_consensus.cc:2749] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:27.013134 13590 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:27.014679 13597 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:27.015352 13590 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:26.969934 13699 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:27.017287 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:27.017971 11976 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:27.018095 13600 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:27.019017 13699 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:26.980685 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:26.953137 13601 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:26.963575 13731 consensus_queue.cc:237] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:27.005134 13662 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:26.989583 13551 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:26.983019 13712 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:26.952911 13735 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:26.985009 13076 raft_consensus.cc:2749] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:26.986986 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:27.019923 11233 raft_consensus.cc:2468] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:27.025347 11380 raft_consensus.cc:2393] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 6: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
I20260321 02:59:26.996017 11307 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:26.987586 11382 raft_consensus.cc:1275] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20260321 02:59:27.027089 11382 pending_rounds.cc:85] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5: Aborting all ops after (but not including) 1
I20260321 02:59:27.027323 11382 pending_rounds.cc:107] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260321 02:59:27.027550 11382 raft_consensus.cc:2889] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:27.027648 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:27.028131 11382 pending_rounds.cc:107] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260321 02:59:27.028440 11382 raft_consensus.cc:2889] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:26.988248 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:26.988530 11384 raft_consensus.cc:1240] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:26.958421 13728 raft_consensus.cc:2804] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:27.029863 13728 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:27.030313 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:27.030385 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:27.030982 11305 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:27.031098 11303 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:27.031416 11308 raft_consensus.cc:1240] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260321 02:59:27.024523 13735 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:27.025142 13712 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:27.024230 13551 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:27.027495 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:27.036092 11310 raft_consensus.cc:2468] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 5.
I20260321 02:59:27.003808 13583 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:26.987737 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:27.038892 11385 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:27.040729 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:26.975255 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 55281.7us
I20260321 02:59:26.963713 13740 consensus_queue.cc:1048] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20260321 02:59:26.995927 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:26.996737 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:26.996488 13669 consensus_queue.cc:237] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:26.952215 13734 consensus_queue.cc:1048] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20260321 02:59:26.997403 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:26.982950 13569 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:26.955116 13579 consensus_queue.cc:1048] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260321 02:59:26.955539 13664 consensus_queue.cc:1048] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260321 02:59:26.971921 13594 consensus_queue.cc:1048] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260321 02:59:27.008531 13659 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:26.984330 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:27.026633 11276 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:26.999502 13721 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:27.043234 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:27.044210 11383 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
W20260321 02:59:27.046267 11942 leader_election.cc:336] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Timed out: RequestConsensusVote RPC to 127.9.213.1:36717 timed out after 1.625s (SENT)
I20260321 02:59:26.966871 13560 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:26.991775 11378 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 6.
I20260321 02:59:27.063509 11942 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:27.053390 11305 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 6.
I20260321 02:59:27.053596 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:27.030262 13728 raft_consensus.cc:3060] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:27.054699 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 82457.7us
I20260321 02:59:27.073282 13721 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:27.074092 13459 raft_consensus.cc:2804] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:27.075336 13560 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:27.080488 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 86471.2us
I20260321 02:59:27.080612 13723 consensus_queue.cc:1048] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:59:27.081700 11200 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:27.082815 11200 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:27.083021 13723 raft_consensus.cc:2804] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:27.083408 13723 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
W20260321 02:59:27.083813 11302 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56630 (request call id 1605) took 2002 ms (client timeout 1725 ms). Trace:
I20260321 02:59:27.083776 13723 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
W20260321 02:59:27.084223 11302 rpcz_store.cc:269] 0321 02:59:25.081593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:25.100919 (+ 19326us) service_pool.cc:225] Handling call
0321 02:59:27.083628 (+1982709us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54001adbd4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:27.083736 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2668479744}
I20260321 02:59:27.086575 11304 raft_consensus.cc:1240] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260321 02:59:27.087384 11304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56616 (request call id 1664) took 1295 ms. Trace:
I20260321 02:59:27.087698 11304 rpcz_store.cc:276] 0321 02:59:25.792497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:26.026546 (+234049us) service_pool.cc:225] Handling call
0321 02:59:27.086119 (+1059573us) spinlock_profiling.cc:243] Waited 2.85 ms on lock 0x7b5800002a98. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b1522680 00007fa8b150cfe9 00007fa8b150ca92 00007fa8aaa8ad34
0321 02:59:27.086327 (+ 208us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:27.087119 (+ 792us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:27.087327 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6562560}
I20260321 02:59:27.089406 11200 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:27.090366 13076 raft_consensus.cc:2804] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:27.090799 13076 raft_consensus.cc:697] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:27.092428 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:27.093245 11232 raft_consensus.cc:2468] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
I20260321 02:59:27.094866 11307 raft_consensus.cc:1275] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260321 02:59:27.095892 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:27.096251 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:27.096843 11230 raft_consensus.cc:2376] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4. Re-sending same reply.
I20260321 02:59:27.091493 13076 consensus_queue.cc:237] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:27.101915 11384 raft_consensus.cc:1275] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260321 02:59:27.102669 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 54278.6us
I20260321 02:59:27.103343 11275 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:27.108106 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:27.108949 11234 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:27.109284 11942 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:27.110884 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:27.111662 11307 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:27.115375 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:27.116149 11232 raft_consensus.cc:2376] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4. Re-sending same reply.
I20260321 02:59:27.116705 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:27.117287 11229 raft_consensus.cc:3060] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:27.128494 11236 raft_consensus.cc:1240] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 4. Current term is 5. Ops: []
W20260321 02:59:27.130888 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 104328us
I20260321 02:59:27.133165 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:27.133113 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:27.134238 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:27.135069 11231 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:27.139329 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:27.062093 13606 raft_consensus.cc:2764] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260321 02:59:27.056305 11972 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:27.075640 13459 raft_consensus.cc:697] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:27.141674 11228 raft_consensus.cc:1275] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260321 02:59:27.141809 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:27.147897 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:27.143584 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:27.151978 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:27.103822 13724 raft_consensus.cc:2749] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:27.129854 13583 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:27.139662 13572 raft_consensus.cc:2804] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:27.139439 13726 raft_consensus.cc:2804] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:27.164322 13726 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:27.164790 13726 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:27.055656 13569 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:27.163596 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:27.163486 13572 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:27.180662 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:27.181509 11378 raft_consensus.cc:2376] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7. Re-sending same reply.
I20260321 02:59:27.139166 13523 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:27.146560 13662 raft_consensus.cc:2804] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:27.139364 13533 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:27.169304 11384 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:27.192322 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:27.199231 11231 raft_consensus.cc:2368] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 6. Current term is 7.
I20260321 02:59:27.200460 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:27.202612 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:27.203363 11236 raft_consensus.cc:2468] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:27.206158 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:27.207252 11384 raft_consensus.cc:2393] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 6: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:27.210604 11307 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 6.
I20260321 02:59:27.212726 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:27.213989 11229 raft_consensus.cc:2468] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:27.214354 11384 raft_consensus.cc:1275] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:27.221447 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:27.226999 11306 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:27.225485 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:27.159677 11972 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45078 (request call id 455) took 1169 ms. Trace:
I20260321 02:59:27.228133 11377 raft_consensus.cc:2393] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 6: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:27.228739 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:27.154006 13513 consensus_queue.cc:1059] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20260321 02:59:27.194288 13572 raft_consensus.cc:3060] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:27.180179 13264 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:27.236043 11378 raft_consensus.cc:1275] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260321 02:59:27.243772 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:27.237470 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:27.239764 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:27.197173 13662 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:27.246485 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:27.247354 11380 raft_consensus.cc:2468] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
W20260321 02:59:27.230862 13725 log.cc:927] Time spent T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260321 02:59:27.219349 13579 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:27.273921 11384 raft_consensus.cc:1275] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260321 02:59:27.272224 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:27.250509 11307 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:27.284898 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:27.291707 11376 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:27.230501 13664 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:27.220136 13523 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:27.220036 13533 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:27.271723 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:27.253141 13662 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:27.289609 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:27.319473 11381 raft_consensus.cc:3060] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:27.308802 13549 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:27.287648 11306 raft_consensus.cc:1275] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260321 02:59:27.310983 11377 raft_consensus.cc:2410] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260321 02:59:27.311045 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:27.322547 11380 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
W20260321 02:59:27.287050 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 155352us
I20260321 02:59:27.324353 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:27.325637 11232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2d71c397418a46d085c9974230b7f964"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:27.325804 11306 raft_consensus.cc:2468] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:27.326462 11232 raft_consensus.cc:606] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:27.327293 11942 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:27.337121 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:27.339330 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:27.338833 11384 raft_consensus.cc:2393] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 8: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:27.340016 11377 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:27.366396 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:27.369145 11376 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:27.369649 11306 raft_consensus.cc:2393] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:27.369630 11275 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:27.345230 11305 raft_consensus.cc:1275] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260321 02:59:27.367404 11304 raft_consensus.cc:1275] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:27.367827 11942 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:27.370431 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:27.368664 11349 leader_election.cc:304] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:27.371230 11376 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 7
I20260321 02:59:27.371984 11349 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:27.230598 11972 rpcz_store.cc:276] 0321 02:59:25.989867 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:26.012991 (+ 23124us) service_pool.cc:225] Handling call
0321 02:59:26.996577 (+983586us) spinlock_profiling.cc:243] Waited 20.3 ms on lock 0x7b58002c3798. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a94791ce 000000000035a3c9 00007fa8b14978e0 00007fa8b1560784 00007fa8b15607f1 00007fa8b150e760 00007fa8b150e044 00007fa8aaa8f834 00007fa8aaa8f7c2
0321 02:59:27.154247 (+157670us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":46669696}
W20260321 02:59:27.330672 13401 consensus_meta.cc:287] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.564s user 0.007s sys 0.007s
I20260321 02:59:27.374042 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:27.310616 13740 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:27.371482 11275 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:27.370373 11305 pending_rounds.cc:85] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da: Aborting all ops after (but not including) 2
I20260321 02:59:27.375555 11349 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:27.378968 11307 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:27.379277 11305 pending_rounds.cc:107] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260321 02:59:27.379498 11379 raft_consensus.cc:2393] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 8: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:27.379622 11305 raft_consensus.cc:2889] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:27.383332 11349 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:27.376634 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:27.385217 11306 raft_consensus.cc:3060] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:27.388085 11349 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:27.306596 13579 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:27.391072 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:27.272104 13459 consensus_queue.cc:237] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:27.311757 13664 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:27.395913 13752 log.cc:927] Time spent T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
W20260321 02:59:27.399904 13747 log.cc:927] Time spent T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.183s user 0.000s sys 0.001s
I20260321 02:59:27.402417 11200 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:27.376272 13761 log.cc:927] Time spent T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.051s user 0.000s sys 0.001s
I20260321 02:59:27.376698 13424 raft_consensus.cc:2749] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260321 02:59:27.373070 13606 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:27.412405 11232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:27.435199 11232 raft_consensus.cc:606] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:27.435662 11232 raft_consensus.cc:614] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260321 02:59:27.414949 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:27.417788 13744 log.cc:927] Time spent T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260321 02:59:27.408509 13656 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:27.418123 13709 raft_consensus.cc:2749] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:27.376082 13551 raft_consensus.cc:2749] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260321 02:59:27.419965 13758 log.cc:927] Time spent T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20260321 02:59:27.469195 11234 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:27.470268 11227 raft_consensus.cc:1240] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 7. Current term is 8. Ops: [3.2-7.4]
I20260321 02:59:27.470852 11235 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 6.
I20260321 02:59:27.420660 11201 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:27.384944 13740 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:27.408834 13590 raft_consensus.cc:2804] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:27.375859 13755 raft_consensus.cc:2804] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:27.385110 13322 raft_consensus.cc:2804] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:27.375823 13669 consensus_queue.cc:1048] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.049s
I20260321 02:59:27.381870 13731 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:27.390461 13659 raft_consensus.cc:2804] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:27.430213 11306 raft_consensus.cc:2468] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:27.499614 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:27.500209 11306 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
W20260321 02:59:27.400487 13746 log.cc:927] Time spent T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20260321 02:59:27.392267 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:27.476056 13722 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:27.437192 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 65763.3us
I20260321 02:59:27.479245 13567 raft_consensus.cc:3055] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:27.438486 11307 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:27.513942 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:27.514680 11307 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:27.448856 11376 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:27.517191 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:27.518076 11376 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:27.454138 13723 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:27.454984 13597 raft_consensus.cc:2749] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:27.457383 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:27.527500 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:27.457598 11381 raft_consensus.cc:2468] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:27.459692 11380 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:27.404830 13358 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:27.474785 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 72428.2us
I20260321 02:59:27.493762 13755 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:27.431883 13569 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:27.387162 13401 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:27.468842 13656 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:27.464380 13606 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:27.417213 11349 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:27.502418 11384 raft_consensus.cc:2393] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 6: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:27.503356 11234 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:27.552836 11234 raft_consensus.cc:606] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:27.479087 13553 raft_consensus.cc:2804] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:27.381855 13754 raft_consensus.cc:2804] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:27.481478 13590 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:27.508193 11275 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:27.559912 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 51789us
I20260321 02:59:27.481807 13718 raft_consensus.cc:2804] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:27.561820 11383 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:27.535157 11306 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:27.536394 11307 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:27.561295 11275 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:27.564018 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:27.540668 11200 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:27.543699 11376 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 6.
I20260321 02:59:27.366719 13735 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:27.500821 13659 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:27.550374 11229 raft_consensus.cc:1240] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260321 02:59:27.541172 13763 log.cc:927] Time spent T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.142s user 0.001s sys 0.000s
I20260321 02:59:27.554493 11377 raft_consensus.cc:1240] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
W20260321 02:59:27.555124 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 137992us
I20260321 02:59:27.580526 11349 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:27.507717 13731 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:27.544872 13749 log.cc:927] Time spent T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.292s user 0.000s sys 0.001s
I20260321 02:59:27.564121 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:27.597273 11307 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:27.555076 13553 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:27.564824 11306 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:27.543079 13755 raft_consensus.cc:3060] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:27.562069 13590 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:27.577615 11200 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:27.630302 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:27.631009 11376 raft_consensus.cc:1240] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: [7.4-7.4]
W20260321 02:59:27.631069 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 53570.2us
I20260321 02:59:27.632229 11200 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:27.632483 11311 raft_consensus.cc:1217] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Deduplicated request from leader. Original: 1.1->[4.2-8.4] Dedup: 4.2->[6.3-8.4]
I20260321 02:59:27.633224 11311 pending_rounds.cc:85] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da: Aborting all ops after (but not including) 2
I20260321 02:59:27.633543 11311 pending_rounds.cc:107] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260321 02:59:27.633847 11311 raft_consensus.cc:2889] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:27.634137 11306 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:27.567847 13569 raft_consensus.cc:2749] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:27.581981 11349 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:27.513109 13567 raft_consensus.cc:740] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:27.589618 11275 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:27.643620 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 54097.3us
I20260321 02:59:27.644888 11275 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:27.475145 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 54565.3us
I20260321 02:59:27.605830 11307 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:27.556806 13445 raft_consensus.cc:2749] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:27.554379 13533 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:27.568312 13600 consensus_meta.cc:287] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.550s user 0.010s sys 0.003s
I20260321 02:59:27.527534 13722 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:27.557930 13754 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:27.559130 13549 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:27.640228 11200 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:27.569696 13718 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:27.579527 13659 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:27.578531 13703 raft_consensus.cc:2749] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:27.646739 11307 raft_consensus.cc:1240] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:27.594761 13751 raft_consensus.cc:2804] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:27.597113 13709 raft_consensus.cc:2804] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260321 02:59:27.654443 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 72777.5us
I20260321 02:59:27.655143 11228 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:27.665732 11228 raft_consensus.cc:606] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:27.604295 13139 consensus_queue.cc:1048] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
I20260321 02:59:27.653149 13579 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:27.593297 13560 consensus_queue.cc:1048] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.072s
I20260321 02:59:27.659008 13076 raft_consensus.cc:2804] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:27.659020 11200 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:27.660425 11382 raft_consensus.cc:2410] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260321 02:59:27.636214 13724 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:27.649621 13712 raft_consensus.cc:2749] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:27.611405 13553 raft_consensus.cc:3060] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:27.667557 13606 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:27.638545 13322 raft_consensus.cc:697] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:27.669929 11349 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:27.661499 13718 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:27.665165 13751 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:27.664913 13709 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:27.677009 13656 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:27.657924 13564 raft_consensus.cc:2749] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:27.698755 13076 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:27.721866 13765 consensus_queue.cc:1048] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260321 02:59:27.722466 13076 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:27.724243 13750 consensus_queue.cc:1048] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20260321 02:59:27.725914 13579 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:27.726990 13579 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:27.728722 13322 consensus_queue.cc:237] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:27.659885 13754 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:27.665011 13445 raft_consensus.cc:2749] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:27.699692 11200 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:27.715346 11236 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:27.733970 11236 raft_consensus.cc:606] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:27.716321 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 46478.1us
I20260321 02:59:27.735762 11349 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:27.722064 13606 raft_consensus.cc:2804] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:27.736071 13076 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:27.736752 13606 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:27.741092 13606 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:27.741923 13753 raft_consensus.cc:2749] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:27.665588 13664 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:27.727710 13767 log.cc:927] Time spent T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
W20260321 02:59:27.704130 13757 log.cc:927] Time spent T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.308s user 0.002s sys 0.000s
I20260321 02:59:27.742134 13076 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:27.762694 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 63150.8us
I20260321 02:59:27.759459 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260321 02:59:27.795882 11200 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:27.700763 13724 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:27.701086 13595 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:27.796976 11200 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:27.703897 13703 raft_consensus.cc:2804] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:27.881791 13703 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:27.881923 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 85039us
I20260321 02:59:27.882360 13703 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:27.882781 11200 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:27.881841 13664 raft_consensus.cc:2804] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:27.883795 11200 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:27.884222 13664 raft_consensus.cc:697] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:27.884824 13736 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:27.885365 13736 raft_consensus.cc:697] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:27.885202 13664 consensus_queue.cc:237] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:27.886199 13736 consensus_queue.cc:237] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:27.747067 13726 consensus_meta.cc:287] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 0.582s user 0.010s sys 0.000s
W20260321 02:59:27.799595 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 421446us
I20260321 02:59:27.762456 13723 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:27.758678 13401 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:27.774641 13523 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:27.771023 13759 log.cc:927] Time spent T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.395s user 0.003s sys 0.000s
I20260321 02:59:27.745972 13597 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:27.742416 13709 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:27.716455 13751 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:27.747548 13662 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:27.810676 13724 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
W20260321 02:59:27.800465 13601 consensus_meta.cc:287] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.780s user 0.008s sys 0.004s
I20260321 02:59:27.811568 13739 raft_consensus.cc:2804] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:27.706246 13567 consensus_queue.cc:260] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:27.765101 13606 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:27.789202 13712 raft_consensus.cc:2749] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:27.757160 13731 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:27.883551 13516 raft_consensus.cc:2804] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election won for term 6
W20260321 02:59:27.883170 13572 consensus_meta.cc:287] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.609s user 0.018s sys 0.012s
I20260321 02:59:27.779398 13740 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:27.896126 13728 consensus_meta.cc:287] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.814s user 0.009s sys 0.008s
I20260321 02:59:27.893981 11200 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:27.957070 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 312245us
I20260321 02:59:27.888926 13595 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:27.962096 11303 raft_consensus.cc:1240] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:27.984162 11942 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:27.988590 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 94667us
I20260321 02:59:27.989602 11200 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:27.992661 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:27.992830 11275 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:27.994431 11275 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
W20260321 02:59:28.014365 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:28.114562 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 120226us
I20260321 02:59:28.115720 11275 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:28.118801 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 117476us
I20260321 02:59:28.124145 11275 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:27.894529 13726 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:27.939257 13735 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:27.961887 13777 log.cc:927] Time spent T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.062s user 0.000s sys 0.001s
I20260321 02:59:28.129897 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:27.938773 13579 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:28.206772 11977 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:27.955989 13739 raft_consensus.cc:697] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:28.202724 11972 raft_consensus.cc:1240] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:28.196580 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:28.230626 11305 raft_consensus.cc:2368] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 6. Current term is 7.
I20260321 02:59:28.141131 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:28.232014 11309 raft_consensus.cc:2368] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 6.
I20260321 02:59:28.209520 11385 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45634 (request call id 323) took 1164 ms. Trace:
I20260321 02:59:28.236415 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:27.940752 13601 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:28.228519 11306 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s user 0.064s sys 0.134s
W20260321 02:59:28.241696 11306 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.546s user 0.064s sys 0.135s
W20260321 02:59:28.189613 13590 consensus_meta.cc:287] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.539s user 0.019s sys 0.007s
I20260321 02:59:28.110556 13516 raft_consensus.cc:697] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:28.126556 13654 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:28.232715 11385 rpcz_store.cc:276] 0321 02:59:27.045318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:27.049559 (+ 4241us) service_pool.cc:225] Handling call
0321 02:59:27.049862 (+ 303us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:27.050201 (+ 339us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:27.050236 (+ 35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:27.050686 (+ 450us) log.cc:844] Serialized 26 byte log entry
0321 02:59:27.234202 (+183516us) raft_consensus.cc:1645] Marking committed up to 6
0321 02:59:27.234241 (+ 39us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:27.234318 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:27.770059 (+535741us) spinlock_profiling.cc:243] Waited 7.53 ms on lock 0x7b440002304c. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93c2dc5 00007fa8a93c37e2 00007fa8ac0f757b 00007fa8ac0f2e4f 00007fa8ac0e6c0e 00007fa8ac0e5f56 00007fa8ac0e5b78 00007fa8ac0ef5c1 00007fa8ac0ef579 00007fa8ac0ef509
0321 02:59:28.136688 (+366629us) raft_consensus.cc:1685] finished
0321 02:59:28.136711 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:28.209418 (+ 72707us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1357,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":7088,"spinlock_wait_cycles":17323904,"thread_start_us":183207,"threads_started":1,"wal-append.queue_time_us":263258}
I20260321 02:59:28.266208 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:27.987661 13572 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.135670 13723 raft_consensus.cc:2804] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
W20260321 02:59:28.224364 11377 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.071s sys 0.130s
W20260321 02:59:28.276559 11377 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s user 0.072s sys 0.130s
I20260321 02:59:28.137194 13753 raft_consensus.cc:2749] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:28.146647 13264 raft_consensus.cc:2804] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:28.146330 13567 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:28.147060 13560 raft_consensus.cc:2804] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:28.126647 13774 raft_consensus.cc:2749] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:28.238059 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:28.186399 13754 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.190328 13722 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:28.191030 13750 raft_consensus.cc:2804] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:28.237098 11304 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:28.127290 13731 consensus_queue.cc:1059] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.124s
I20260321 02:59:28.198612 13726 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:28.266892 11385 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 6.
I20260321 02:59:28.310914 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:28.311321 11942 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:28.282980 13264 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:28.136895 13669 consensus_queue.cc:1048] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.123s
I20260321 02:59:28.320453 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:27.951560 13597 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.144865 13740 consensus_queue.cc:1048] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.120s
W20260321 02:59:28.281906 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 292394us
I20260321 02:59:28.112156 13728 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:28.287830 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:28.262399 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:28.327708 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:28.329515 11305 raft_consensus.cc:1275] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260321 02:59:28.127136 13139 consensus_queue.cc:1059] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.117s
W20260321 02:59:28.284389 13785 log.cc:927] Time spent T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.305s user 0.001s sys 0.000s
I20260321 02:59:28.321520 11302 raft_consensus.cc:1240] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:28.352469 13754 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:28.353492 13728 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:28.260696 13590 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.274683 13654 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.285102 13723 raft_consensus.cc:697] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:28.292140 13560 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:28.347988 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:28.326295 13774 raft_consensus.cc:2804] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:28.348366 13662 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:28.320281 13264 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:28.350353 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:28.321120 13750 raft_consensus.cc:697] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:28.349200 13733 consensus_queue.cc:1048] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260321 02:59:28.360875 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:28.362558 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:28.349449 13579 raft_consensus.cc:3055] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Stepping down as leader of term 6
W20260321 02:59:28.324940 13755 consensus_meta.cc:287] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 0.675s user 0.013s sys 0.000s
I20260321 02:59:28.366142 11304 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:28.375352 13723 consensus_queue.cc:237] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.379114 13654 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:28.372875 13560 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:28.320740 13739 consensus_queue.cc:237] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.343087 13712 consensus_queue.cc:1048] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260321 02:59:28.355686 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:28.387416 13750 consensus_queue.cc:237] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.389264 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:28.403213 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:28.403424 11306 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:28.405325 11275 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:28.392850 11305 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
W20260321 02:59:28.395488 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 84282.6us
I20260321 02:59:28.406658 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:28.407363 11305 raft_consensus.cc:2393] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 6: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:28.365808 13606 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:28.373409 13774 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:28.401983 11380 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.028s sys 0.039s
I20260321 02:59:28.342614 13516 consensus_queue.cc:237] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.407073 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:28.350595 13717 consensus_queue.cc:1059] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.036s
I20260321 02:59:28.408365 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:28.417006 11304 raft_consensus.cc:2393] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 8: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:28.408849 11236 raft_consensus.cc:1275] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260321 02:59:28.393563 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:28.410449 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:28.418771 11305 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:28.420185 11228 raft_consensus.cc:1275] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260321 02:59:28.421842 13715 raft_consensus.cc:3055] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:28.422274 13715 raft_consensus.cc:740] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:28.423085 13715 consensus_queue.cc:260] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:28.424801 13590 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:28.411917 11380 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.531s user 0.029s sys 0.039s
I20260321 02:59:28.395222 13579 raft_consensus.cc:740] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:28.400216 13755 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:28.413661 13774 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:28.413573 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:28.413944 11306 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 7.
I20260321 02:59:28.426651 11229 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58214 (request call id 1290) took 1142 ms. Trace:
I20260321 02:59:28.433832 11229 rpcz_store.cc:276] 0321 02:59:27.284095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:27.284279 (+ 184us) service_pool.cc:225] Handling call
0321 02:59:27.284805 (+ 526us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:27.570997 (+286192us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:28.426366 (+855369us) spinlock_profiling.cc:243] Waited 265 ms on lock 0x7b5400060c54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:28.426564 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":610068864}
I20260321 02:59:28.434706 11305 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:28.427482 11972 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45078 (request call id 463) took 1005 ms. Trace:
I20260321 02:59:28.435369 11972 rpcz_store.cc:276] 0321 02:59:27.421901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:27.422078 (+ 177us) service_pool.cc:225] Handling call
0321 02:59:27.422415 (+ 337us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:28.229862 (+807447us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:28.427266 (+197404us) spinlock_profiling.cc:243] Waited 780 ms on lock 0x7b540063e554. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:28.427426 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1793793920}
I20260321 02:59:28.436118 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:28.414160 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:28.437279 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:28.437777 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:28.438011 11234 raft_consensus.cc:1240] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 7. Ops: [5.4-5.4]
I20260321 02:59:28.437940 11305 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 5.
I20260321 02:59:28.409432 13654 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:28.429210 13718 consensus_meta.cc:287] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.694s user 0.009s sys 0.002s
I20260321 02:59:28.417332 13662 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election won for term 8
W20260321 02:59:28.433959 13703 consensus_meta.cc:287] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.551s user 0.012s sys 0.003s
I20260321 02:59:28.478675 13703 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.418107 13572 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:28.475733 11973 raft_consensus.cc:1240] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:28.417443 13754 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:28.481076 13754 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.483101 13754 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:28.514950 11380 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1627) took 1063 ms. Trace:
I20260321 02:59:28.515235 13739 consensus_queue.cc:1048] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:59:28.518195 13755 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:28.515409 11380 rpcz_store.cc:276] 0321 02:59:27.451917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:27.607997 (+156080us) service_pool.cc:225] Handling call
0321 02:59:27.649857 (+ 41860us) raft_consensus.cc:1471] Updating replica for 3 ops
0321 02:59:27.704147 (+ 54290us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:27.704178 (+ 31us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0321 02:59:27.707142 (+ 2964us) log.cc:844] Serialized 78 byte log entry
0321 02:59:28.429542 (+722400us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:59:28.429620 (+ 78us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:28.429867 (+ 247us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:28.514723 (+ 84856us) raft_consensus.cc:1685] finished
0321 02:59:28.514744 (+ 21us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:28.514892 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":14351,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"thread_start_us":601292,"threads_started":2,"wal-append.queue_time_us":648884}
I20260321 02:59:28.519382 13712 consensus_queue.cc:1048] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:59:28.520102 11200 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:28.521323 13712 raft_consensus.cc:2804] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:28.521718 13712 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:28.521888 11942 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:28.523612 13755 raft_consensus.cc:2804] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:28.518095 13733 consensus_queue.cc:1048] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:28.520176 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:28.524897 11380 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:28.522079 13712 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:28.468482 13654 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:28.509688 13659 consensus_meta.cc:287] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.808s user 0.012s sys 0.000s
W20260321 02:59:28.526727 13553 consensus_meta.cc:287] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.810s user 0.007s sys 0.003s
I20260321 02:59:28.545243 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:28.531227 13724 consensus_meta.cc:287] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.576s user 0.012s sys 0.000s
I20260321 02:59:28.524076 13755 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:28.472261 13718 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:28.549072 11942 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:28.533390 13715 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:28.537552 13703 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:28.567979 11380 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:28.585868 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:28.525524 13560 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:28.588225 11379 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40950 (request call id 1227) took 1605 ms (client timeout 1502 ms). Trace:
W20260321 02:59:28.589818 11379 rpcz_store.cc:269] 0321 02:59:26.983887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:26.984052 (+ 165us) service_pool.cc:225] Handling call
0321 02:59:28.588021 (+1603969us) spinlock_profiling.cc:243] Waited 303 ms on lock 0x7b54000c3854. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:28.588163 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":696284928}
I20260321 02:59:28.593400 11235 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 6.
I20260321 02:59:28.588526 11378 raft_consensus.cc:1240] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260321 02:59:28.591586 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:28.570246 13709 consensus_meta.cc:287] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.616s user 0.009s sys 0.001s
I20260321 02:59:28.569535 13755 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:28.550191 13662 raft_consensus.cc:697] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:28.608791 13662 consensus_queue.cc:237] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.2, Last appended by leader: 2, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.545897 13659 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.588961 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:28.544839 13322 consensus_queue.cc:1048] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20260321 02:59:28.550045 13669 consensus_queue.cc:1048] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.063s
I20260321 02:59:28.601053 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:28.566272 13724 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.548084 13553 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.568600 13739 consensus_queue.cc:1059] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.036s
I20260321 02:59:28.664872 11309 raft_consensus.cc:1240] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7. Ops: []
W20260321 02:59:28.645850 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 106116us
I20260321 02:59:28.625871 11380 raft_consensus.cc:1275] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260321 02:59:28.631851 11381 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1617) took 1330 ms. Trace:
I20260321 02:59:28.673166 11380 raft_consensus.cc:1275] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260321 02:59:28.673601 13774 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:28.674813 11380 raft_consensus.cc:1240] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:28.675654 13774 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:28.679396 11380 raft_consensus.cc:1240] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: [7.4-7.4]
W20260321 02:59:28.678906 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 943141us
I20260321 02:59:28.568627 13726 raft_consensus.cc:2749] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:28.682679 11200 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:28.683456 11350 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 539835us
I20260321 02:59:28.684547 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:28.686077 11305 raft_consensus.cc:3060] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:28.671862 11381 rpcz_store.cc:276] 0321 02:59:27.301336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:27.528508 (+227172us) service_pool.cc:225] Handling call
0321 02:59:27.529011 (+ 503us) raft_consensus.cc:1471] Updating replica for 3 ops
0321 02:59:27.529430 (+ 419us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:27.529458 (+ 28us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0321 02:59:27.540011 (+ 10553us) log.cc:844] Serialized 78 byte log entry
0321 02:59:27.588696 (+ 48685us) raft_consensus.cc:1645] Marking committed up to 5
0321 02:59:27.713887 (+125191us) log.cc:844] Serialized 14 byte log entry
0321 02:59:27.715434 (+ 1547us) log.cc:844] Serialized 14 byte log entry
0321 02:59:28.613309 (+897875us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b18002c0da0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b2300441 00007fa8b230cca1 00007fa8a942a63a 00007fa8b2aa65c9 00007fa8a92e8c50 00007fa8a94b32b4 00007fa8a94b3225 00007fa8ac544440 00007fa8ac556af6 00007fa8ac567667 00007fa8ac5675e8
0321 02:59:28.613933 (+ 624us) log.cc:844] Serialized 14 byte log entry
0321 02:59:28.614171 (+ 238us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:28.631581 (+ 17410us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:28.631634 (+ 53us) raft_consensus.cc:1685] finished
0321 02:59:28.631651 (+ 17us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:28.631799 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20070,"prepare.queue_time_us":83136,"prepare.run_cpu_time_us":19621,"prepare.run_wall_time_us":71669,"raft.queue_time_us":914501,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":19331,"spinlock_wait_cycles":106807168,"thread_start_us":103754,"threads_started":2,"wal-append.queue_time_us":97824}
I20260321 02:59:28.693236 11200 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:28.695884 11305 raft_consensus.cc:2468] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 6.
I20260321 02:59:28.696955 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:28.697361 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:28.697618 11305 raft_consensus.cc:3060] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
W20260321 02:59:28.631582 13751 consensus_meta.cc:287] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.699s user 0.008s sys 0.004s
I20260321 02:59:28.685526 11377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1628) took 1233 ms. Trace:
I20260321 02:59:28.670521 13401 raft_consensus.cc:2749] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:28.700773 11200 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:28.709604 11349 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:28.701753 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:28.700489 11377 rpcz_store.cc:276] 0321 02:59:27.452347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:27.635559 (+183212us) service_pool.cc:225] Handling call
0321 02:59:27.655680 (+ 20121us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:27.655904 (+ 224us) raft_consensus.cc:1534] Early marking committed up to index 7
0321 02:59:27.661062 (+ 5158us) log.cc:844] Serialized 14 byte log entry
0321 02:59:28.685191 (+1024129us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:28.685246 (+ 55us) raft_consensus.cc:1645] Marking committed up to 7
0321 02:59:28.685289 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:28.685353 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:28.685473 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":408030,"prepare.queue_time_us":13077,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":34,"raft.queue_time_us":418064,"thread_start_us":615747,"threads_started":1,"wal-append.queue_time_us":651782}
I20260321 02:59:28.711371 11381 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:28.702858 11385 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45634 (request call id 327) took 1247 ms. Trace:
I20260321 02:59:28.714434 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:28.713042 11385 rpcz_store.cc:276] 0321 02:59:27.454981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:28.311813 (+856832us) service_pool.cc:225] Handling call
0321 02:59:28.312112 (+ 299us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:28.312349 (+ 237us) raft_consensus.cc:1534] Early marking committed up to index 6
0321 02:59:28.312681 (+ 332us) log.cc:844] Serialized 14 byte log entry
0321 02:59:28.425851 (+113170us) spinlock_profiling.cc:243] Waited 3.47 ms on lock 0x7b18002c0d40. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b2300441 00007fa8b230cca1 00007fa8a9498d17 00007fa8b2276b19 00007fa8a94b546b 00007fa8a94b370f 00007fa8a94b59d2 00007fa8ac0e55ab 00007fa8ac0eb0c3 00007fa8ac0eb5da 00007fa8ac556e0a
0321 02:59:28.702523 (+276672us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:28.702583 (+ 60us) raft_consensus.cc:1645] Marking committed up to 6
0321 02:59:28.702633 (+ 50us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:28.702694 (+ 61us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:28.702806 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":242587,"prepare.queue_time_us":184,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"raft.queue_time_us":254642,"spinlock_wait_cycles":7985920,"thread_start_us":28587,"threads_started":1,"wal-append.queue_time_us":346719}
I20260321 02:59:28.703262 13445 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:28.718606 13401 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:28.704861 13736 raft_consensus.cc:2804] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:28.710127 13703 raft_consensus.cc:2804] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:28.614367 13709 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.689028 13731 consensus_queue.cc:1048] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260321 02:59:28.746706 11385 raft_consensus.cc:1240] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:28.753746 11377 raft_consensus.cc:1275] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260321 02:59:28.715025 13654 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:28.754487 11381 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 7.
I20260321 02:59:28.703596 13264 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:28.710925 13751 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.771106 11379 raft_consensus.cc:2393] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 7: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:28.749400 13736 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:28.772109 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 86520.7us
I20260321 02:59:28.760816 13549 raft_consensus.cc:2804] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:28.722404 13445 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.722219 13401 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.758406 13703 raft_consensus.cc:697] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:28.768750 13659 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:28.776736 13736 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:28.796810 13560 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:28.801851 13264 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:28.802517 13445 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:28.803319 13724 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:28.807968 13553 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:28.720314 13782 consensus_queue.cc:1048] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.058s
W20260321 02:59:28.803864 13798 log.cc:927] Time spent T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.100s user 0.000s sys 0.001s
I20260321 02:59:28.820030 13401 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:28.820960 13755 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:28.789700 13459 consensus_queue.cc:1048] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260321 02:59:28.797909 13718 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:28.814420 13811 consensus_queue.cc:1048] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.024s
I20260321 02:59:28.824965 13579 consensus_queue.cc:260] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.826248 13579 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:28.826323 13703 consensus_queue.cc:237] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.828205 13709 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:28.825042 13751 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:28.836999 13755 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:28.839927 13597 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:28.841634 13712 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.844432 13712 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:28.845012 13790 raft_consensus.cc:3055] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:28.845413 13790 raft_consensus.cc:740] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:28.846210 13790 consensus_queue.cc:260] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.847360 13790 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:28.843655 13601 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:28.857779 13736 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:28.860672 13736 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:28.860715 13764 log.cc:927] Time spent T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 1.314s user 0.005s sys 0.007s
I20260321 02:59:28.858461 13664 raft_consensus.cc:993] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb: : Instructing follower 83b12b883a7f4129a87dadf5a78f08da to start an election
I20260321 02:59:28.863909 13736 raft_consensus.cc:1081] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Signalling peer 83b12b883a7f4129a87dadf5a78f08da to start an election
W20260321 02:59:28.900632 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 360348us
W20260321 02:59:28.901841 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 201031us
I20260321 02:59:28.913239 11306 raft_consensus.cc:1240] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 7. Current term is 8. Ops: [3.2-7.4]
I20260321 02:59:28.915161 11307 raft_consensus.cc:1240] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:28.917158 11310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1622) took 1694 ms. Trace:
I20260321 02:59:28.919872 11303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1634) took 1537 ms. Trace:
I20260321 02:59:28.917824 11310 rpcz_store.cc:276] 0321 02:59:27.222416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:27.222571 (+ 155us) service_pool.cc:225] Handling call
0321 02:59:27.233374 (+ 10803us) raft_consensus.cc:1471] Updating replica for 2 ops
0321 02:59:27.239574 (+ 6200us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:27.239604 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0321 02:59:27.247472 (+ 7868us) log.cc:844] Serialized 52 byte log entry
0321 02:59:27.446425 (+198953us) raft_consensus.cc:1645] Marking committed up to 3
0321 02:59:27.552568 (+106143us) log.cc:844] Serialized 14 byte log entry
0321 02:59:27.552820 (+ 252us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:27.552890 (+ 70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:28.860139 (+1307249us) spinlock_profiling.cc:243] Waited 3.9 ms on lock 0x7b440002304c. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93c2dc5 00007fa8a93c37e2 00007fa8ac0f757b 00007fa8ac0f2e4f 00007fa8ac0e6c0e 00007fa8ac0e5f56 00007fa8ac0e5b78 00007fa8ac0ef5c1 00007fa8ac0ef579 00007fa8ac0ef509
0321 02:59:28.916180 (+ 56041us) raft_consensus.cc:1685] finished
0321 02:59:28.916375 (+ 195us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:28.916850 (+ 475us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19916,"prepare.queue_time_us":137722,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":21201,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":21933,"spinlock_wait_cycles":9422848,"thread_start_us":213943,"threads_started":2,"wal-append.queue_time_us":216330}
I20260321 02:59:28.920389 11303 rpcz_store.cc:276] 0321 02:59:27.385458 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:27.386011 (+ 553us) service_pool.cc:225] Handling call
0321 02:59:27.386394 (+ 383us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:28.105037 (+718643us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:28.919196 (+814159us) spinlock_profiling.cc:243] Waited 575 ms on lock 0x7b54001ac2d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:28.919657 (+ 461us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1323420928}
W20260321 02:59:28.924439 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 60872.2us
I20260321 02:59:28.926337 11349 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:28.930003 11383 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1630) took 1476 ms. Trace:
I20260321 02:59:28.931025 13076 raft_consensus.cc:2804] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:28.931915 13076 raft_consensus.cc:697] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
W20260321 02:59:28.932502 11940 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:28.932672 11384 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1629) took 1479 ms. Trace:
W20260321 02:59:28.934171 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 152048us
I20260321 02:59:28.933568 13076 consensus_queue.cc:237] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:28.938759 11350 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:28.933283 11376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1622) took 1584 ms. Trace:
I20260321 02:59:28.941890 11376 rpcz_store.cc:276] 0321 02:59:27.349140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:27.565178 (+216038us) service_pool.cc:225] Handling call
0321 02:59:27.565601 (+ 423us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:27.631455 (+ 65854us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:28.932611 (+1301156us) spinlock_profiling.cc:243] Waited 43.1 ms on lock 0x7b5400099554. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:28.933077 (+ 466us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":99062016}
I20260321 02:59:28.935568 11942 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:28.944501 13662 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:28.933275 11384 rpcz_store.cc:276] 0321 02:59:27.452676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:27.714852 (+262176us) service_pool.cc:225] Handling call
0321 02:59:27.731506 (+ 16654us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:28.631897 (+900391us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:28.631927 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:28.632508 (+ 581us) log.cc:844] Serialized 26 byte log entry
0321 02:59:28.757297 (+124789us) raft_consensus.cc:1645] Marking committed up to 4
0321 02:59:28.757333 (+ 36us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:28.758840 (+ 1507us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:28.932144 (+173304us) raft_consensus.cc:1685] finished
0321 02:59:28.932180 (+ 36us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:28.932461 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":921558,"raft.queue_time_us":923759,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":25527,"thread_start_us":63198,"threads_started":1,"wal-append.queue_time_us":140967}
I20260321 02:59:28.945680 13662 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:28.930565 11383 rpcz_store.cc:276] 0321 02:59:27.452972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:27.985437 (+532465us) service_pool.cc:225] Handling call
0321 02:59:28.146097 (+160660us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:28.698420 (+552323us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:28.698464 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:28.699014 (+ 550us) log.cc:844] Serialized 26 byte log entry
0321 02:59:28.928926 (+229912us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:28.929047 (+ 121us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:28.929241 (+ 194us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:28.929362 (+ 121us) raft_consensus.cc:1685] finished
0321 02:59:28.929386 (+ 24us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:28.929719 (+ 333us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":544871,"raft.queue_time_us":544114,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":9575,"thread_start_us":177870,"threads_started":1,"wal-append.queue_time_us":99556}
I20260321 02:59:28.951421 13709 raft_consensus.cc:2764] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260321 02:59:28.964784 11302 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1642) took 1483 ms. Trace:
I20260321 02:59:28.989159 11302 rpcz_store.cc:276] 0321 02:59:27.481760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:27.652461 (+170701us) service_pool.cc:225] Handling call
0321 02:59:27.667102 (+ 14641us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:28.351450 (+684348us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:28.964124 (+612674us) spinlock_profiling.cc:243] Waited 654 ms on lock 0x7b54002d0854. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:28.964584 (+ 460us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1504488320}
I20260321 02:59:28.966506 11378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1615) took 1712 ms. Trace:
I20260321 02:59:28.997591 11309 raft_consensus.cc:1275] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260321 02:59:28.951786 11376 raft_consensus.cc:1275] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:28.991415 11310 raft_consensus.cc:1275] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260321 02:59:28.999544 13779 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:28.991330 11302 raft_consensus.cc:1240] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260321 02:59:28.947781 13703 consensus_queue.cc:1059] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:59:28.950783 13600 raft_consensus.cc:2749] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:28.986578 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:28.944540 13553 consensus_queue.cc:1059] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260321 02:59:28.991868 11378 rpcz_store.cc:276] 0321 02:59:27.254000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:27.287716 (+ 33716us) service_pool.cc:225] Handling call
0321 02:59:28.588189 (+1300473us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:28.594597 (+ 6408us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:28.966016 (+371419us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54000c3850. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54bc8d 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:28.966304 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2926954752}
I20260321 02:59:28.983927 13662 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:29.036173 13505 raft_consensus.cc:2804] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:29.036499 13779 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.039011 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:29.000888 11310 raft_consensus.cc:1275] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260321 02:59:29.064846 13790 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:29.065954 13790 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:28.985476 13712 consensus_queue.cc:1059] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.084s
I20260321 02:59:29.070495 13712 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:29.071062 13553 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:29.051893 11310 pending_rounds.cc:85] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da: Aborting all ops after (but not including) 2
I20260321 02:59:29.073179 11310 pending_rounds.cc:107] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260321 02:59:29.078205 13717 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:29.078029 13553 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:29.079011 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 153098us
I20260321 02:59:29.079288 13717 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.080236 11349 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:29.082142 13789 raft_consensus.cc:3055] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:29.082666 13789 raft_consensus.cc:740] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:29.083513 11349 leader_election.cc:400] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267) with higher term. Message: Invalid argument: T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 6. Current term is 7.
I20260321 02:59:29.084483 13799 raft_consensus.cc:3055] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:29.085309 11349 leader_election.cc:403] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260321 02:59:29.076992 11310 raft_consensus.cc:2889] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:29.063567 13736 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:29.087364 13799 raft_consensus.cc:740] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:29.081200 13553 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:29.088310 13505 raft_consensus.cc:697] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:29.092849 13600 raft_consensus.cc:2749] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:29.100834 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:29.100646 13736 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.101581 13601 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:29.110229 11942 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:29.111838 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:29.114296 11310 raft_consensus.cc:1275] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:29.071748 13712 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.124516 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:29.091445 13724 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:29.125877 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:29.125867 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:29.128243 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:29.128774 13814 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:29.130026 13590 raft_consensus.cc:2749] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267) with higher term. Message: Invalid argument: T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 6. Current term is 7.
I20260321 02:59:29.111953 13718 raft_consensus.cc:3055] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:29.093848 13734 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:29.147639 13819 raft_consensus.cc:2804] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader election won for term 6
W20260321 02:59:29.157251 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:29.147620 13551 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:29.145917 13718 raft_consensus.cc:740] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:29.177996 11310 raft_consensus.cc:1240] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:29.169152 13600 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:29.179756 11972 raft_consensus.cc:2410] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260321 02:59:29.132107 13505 consensus_queue.cc:237] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:29.169245 13703 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:29.170091 13590 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:29.182026 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:29.169718 13662 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:29.185806 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:29.183259 13600 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.116108 13789 consensus_queue.cc:260] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.188959 13551 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:29.191448 13825 log.cc:927] Time spent T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.092s user 0.000s sys 0.001s
I20260321 02:59:29.192152 13564 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:29.189191 13724 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.185695 13819 raft_consensus.cc:697] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:29.193082 13662 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.194294 13819 consensus_queue.cc:237] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:29.194891 13724 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:29.195544 13662 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:29.185447 13601 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.183528 13703 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:29.192183 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:29.198199 13718 consensus_queue.cc:260] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.190781 13814 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.187923 13715 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:29.203235 13551 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:29.203343 13789 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:29.206998 11307 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1645) took 1694 ms. Trace:
I20260321 02:59:29.207513 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:29.187891 13734 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.210889 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:29.211109 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:29.215490 11382 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40934 (request call id 1610) took 1990 ms (client timeout 1599 ms). Trace:
I20260321 02:59:29.216267 13814 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:29.216182 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:29.216683 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:29.188315 13717 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:29.183683 13590 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:29.223374 11303 raft_consensus.cc:1240] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:29.223599 13734 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:29.218623 11382 rpcz_store.cc:269] 0321 02:59:27.225142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:27.226070 (+ 928us) service_pool.cc:225] Handling call
0321 02:59:29.214891 (+1988821us) spinlock_profiling.cc:243] Waited 429 ms on lock 0x7b540030f554. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:29.215184 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":986020096}
I20260321 02:59:29.226114 13590 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:29.228271 11311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1637) took 1796 ms. Trace:
I20260321 02:59:29.228852 11311 rpcz_store.cc:276] 0321 02:59:27.431975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:27.631641 (+199666us) service_pool.cc:225] Handling call
0321 02:59:27.632232 (+ 591us) raft_consensus.cc:1471] Updating replica for 3 ops
0321 02:59:27.656291 (+ 24059us) raft_consensus.cc:1534] Early marking committed up to index 1
0321 02:59:27.656321 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0321 02:59:27.656860 (+ 539us) log.cc:844] Serialized 52 byte log entry
0321 02:59:28.956711 (+1299851us) spinlock_profiling.cc:243] Waited 1.51 ms on lock 0x7b1800274c00. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b2300441 00007fa8b230cca1 00007fa8a9498d17 00007fa8b2276b19 00007fa8a94b546b 00007fa8a94b370f 00007fa8a94b59d2 00007fa8ac0e55ab 00007fa8ac0eb0c3 00007fa8ac0eb471 00007fa8ac507def
0321 02:59:29.169391 (+212680us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:59:29.169603 (+ 212us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:29.179895 (+ 10292us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:29.227682 (+ 47787us) raft_consensus.cc:1685] finished
0321 02:59:29.227708 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:29.228055 (+ 347us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18782,"raft.queue_time_us":19124,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":3587,"spinlock_wait_cycles":3478912,"thread_start_us":99781,"threads_started":1,"wal-append.queue_time_us":1530262}
I20260321 02:59:29.196291 13784 raft_consensus.cc:493] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:29.238511 13715 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:29.240391 11305 consensus_meta.cc:287] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.542s user 0.010s sys 0.021s
I20260321 02:59:29.240955 11305 raft_consensus.cc:2468] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:29.217994 11307 rpcz_store.cc:276] 0321 02:59:27.512556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:27.662850 (+150294us) service_pool.cc:225] Handling call
0321 02:59:27.663121 (+ 271us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:28.915749 (+1252628us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:29.206655 (+290906us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54001d56d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:29.206929 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2877864832}
I20260321 02:59:29.215549 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:29.221676 13790 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:29.249054 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:29.261951 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:29.263824 11232 raft_consensus.cc:3060] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
W20260321 02:59:29.265291 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 182066us
I20260321 02:59:29.267645 11973 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45088 (request call id 285) took 1138 ms. Trace:
I20260321 02:59:29.272339 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:59:29.273363 13784 raft_consensus.cc:515] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:29.274097 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:29.276537 13736 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:29.278623 13600 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:29.279796 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:29.280828 11234 raft_consensus.cc:2393] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 5: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:29.280732 13601 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:29.196424 13799 consensus_queue.cc:260] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.281489 13579 consensus_queue.cc:1059] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:59:29.282419 13703 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:29.281786 11133 catalog_manager.cc:5671] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 2 to 5, leader changed from 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 5 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:59:29.283192 13765 raft_consensus.cc:3055] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:29.283758 13765 raft_consensus.cc:740] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:29.284694 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:29.285053 13765 consensus_queue.cc:260] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.278715 13715 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:29.286186 11201 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:29.284013 11133 catalog_manager.cc:5671] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:59:29.283324 13718 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:29.288481 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:29.289417 11310 raft_consensus.cc:2393] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:29.288887 11133 catalog_manager.cc:5671] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 1 to 2, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4). New cstate: current_term: 2 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:59:29.289754 13593 consensus_queue.cc:1059] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260321 02:59:29.291218 13593 raft_consensus.cc:2749] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:29.291828 13724 raft_consensus.cc:3055] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:29.292794 13724 raft_consensus.cc:740] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:29.269034 13712 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:29.294829 13724 consensus_queue.cc:260] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:29.296774 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:29.297617 11310 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:29.298141 11349 leader_election.cc:400] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Vote denied by peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267) with higher term. Message: Invalid argument: T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 6.
I20260321 02:59:29.298668 11349 leader_election.cc:403] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260321 02:59:29.268194 11973 rpcz_store.cc:276] 0321 02:59:28.128921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:28.137364 (+ 8443us) service_pool.cc:225] Handling call
0321 02:59:28.242750 (+105386us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:28.480412 (+237662us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:29.267051 (+786639us) spinlock_profiling.cc:243] Waited 214 ms on lock 0x7b540046dcd4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:29.267447 (+ 396us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":492419840}
I20260321 02:59:29.296967 13724 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:29.302158 11349 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:29.291432 11133 catalog_manager.cc:5671] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 3 to 4, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4). New cstate: current_term: 4 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:59:29.314320 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:29.316713 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:29.317560 11302 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:29.302954 13601 raft_consensus.cc:2749] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267) with higher term. Message: Invalid argument: T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 6.
W20260321 02:59:29.328091 13829 log.cc:927] Time spent T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260321 02:59:29.332307 13600 raft_consensus.cc:2749] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:29.328711 11307 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2d71c397418a46d085c9974230b7f964"
dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:56630
I20260321 02:59:29.333760 11307 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:59:29.334261 11307 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:29.337251 11349 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:29.339752 13600 raft_consensus.cc:2804] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:29.339579 11349 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:29.364629 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 144363us
W20260321 02:59:29.368686 13832 log.cc:927] Time spent T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20260321 02:59:29.331817 13784 leader_election.cc:290] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:29.343315 13600 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
W20260321 02:59:29.412460 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 124720us
W20260321 02:59:29.415503 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 114321us
I20260321 02:59:29.412214 13600 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:29.379292 13601 raft_consensus.cc:2749] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:29.417167 13662 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:29.440407 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:29.443150 13734 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:29.443711 13734 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.445375 13560 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:29.445958 13734 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:29.446029 13560 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:29.448388 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 108970us
I20260321 02:59:29.450294 13424 consensus_queue.cc:1048] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260321 02:59:29.450711 11349 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:29.478036 11302 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:29.491405 11232 raft_consensus.cc:2468] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:29.493654 13595 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:29.506129 13601 raft_consensus.cc:2804] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:29.506796 13601 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:29.507321 13601 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:29.516542 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:29.517439 11235 raft_consensus.cc:2410] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260321 02:59:29.514833 11234 raft_consensus.cc:1240] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:29.504648 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:29.518703 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:29.488999 13662 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:29.522991 13560 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:29.527242 11227 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58200 (request call id 1644) took 1053 ms. Trace:
I20260321 02:59:29.527637 11227 rpcz_store.cc:276] 0321 02:59:28.474067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:28.474201 (+ 134us) service_pool.cc:225] Handling call
0321 02:59:29.499557 (+1025356us) spinlock_profiling.cc:243] Waited 8.67 ms on lock 0x7b5800111398. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b150e760 00007fa8b150e044 00007fa8aaa8f834 00007fa8aaa8f7c2
0321 02:59:29.527164 (+ 27607us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":19950464}
W20260321 02:59:29.536160 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:29.520182 13630 consensus_queue.cc:1048] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
W20260321 02:59:29.519898 13837 log.cc:927] Time spent T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20260321 02:59:29.539772 13736 consensus_queue.cc:1059] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:59:29.540719 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:29.543226 11200 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:29.543978 13594 consensus_queue.cc:1059] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:59:29.544371 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:29.544766 13805 consensus_queue.cc:1048] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:29.545229 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:29.545454 13790 consensus_queue.cc:1048] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:59:29.546041 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:29.546075 13717 consensus_queue.cc:1059] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:59:29.548884 13513 consensus_queue.cc:1048] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260321 02:59:29.551435 13595 consensus_queue.cc:1059] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260321 02:59:29.558667 11233 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:29.586969 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:29.560681 13662 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:29.589936 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:29.561638 13736 raft_consensus.cc:3055] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:29.590766 13736 raft_consensus.cc:740] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:29.591924 13736 consensus_queue.cc:260] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.593300 13736 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:29.594333 13595 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:29.594946 13595 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.560379 13600 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:29.597118 13595 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:29.598686 13600 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:29.605998 13765 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:29.605723 11307 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.614022 13549 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:29.614894 13549 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:29.615571 11307 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:29.568676 13594 consensus_queue.cc:1059] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 2, Time since last communication: 0.049s
I20260321 02:59:29.618629 13594 raft_consensus.cc:3055] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:29.619243 13594 raft_consensus.cc:740] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:29.576707 13424 raft_consensus.cc:3055] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:29.575428 13717 raft_consensus.cc:3055] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:29.620502 13594 consensus_queue.cc:260] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.621206 13717 raft_consensus.cc:740] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
W20260321 02:59:29.579536 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:29.584712 11306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56616 (request call id 1688) took 1207 ms. Trace:
I20260321 02:59:29.573303 11349 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:29.623139 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 50078.8us
I20260321 02:59:29.582026 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:29.564543 13723 raft_consensus.cc:2804] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:29.625773 13723 raft_consensus.cc:697] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:29.627118 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:29.580061 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:29.562021 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:29.620718 13424 raft_consensus.cc:740] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:29.573376 11231 raft_consensus.cc:2410] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260321 02:59:29.630064 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 224568us
I20260321 02:59:29.572299 11236 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:29.586349 11310 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:29.626856 13723 consensus_queue.cc:237] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.574153 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:29.622812 11306 rpcz_store.cc:276] 0321 02:59:28.377366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:28.433935 (+ 56569us) service_pool.cc:225] Handling call
0321 02:59:28.434275 (+ 340us) raft_consensus.cc:1471] Updating replica for 3 ops
0321 02:59:28.914272 (+479997us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:29.584178 (+669906us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x7b54000dfa54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:29.584583 (+ 405us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1098706176}
I20260321 02:59:29.641047 13782 raft_consensus.cc:2804] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:29.641372 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:29.642110 13782 raft_consensus.cc:697] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
W20260321 02:59:29.642879 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:29.671494 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:29.675714 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:29.677562 13424 consensus_queue.cc:260] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:29.571825 13719 raft_consensus.cc:3055] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:29.679975 13719 raft_consensus.cc:740] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:29.681361 13719 consensus_queue.cc:260] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.622268 13717 consensus_queue.cc:260] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:29.685802 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:29.643250 13782 consensus_queue.cc:237] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.555940 13715 consensus_queue.cc:1059] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.134s
I20260321 02:59:29.702926 11236 raft_consensus.cc:1240] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260321 02:59:29.704782 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:29.705744 11236 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 6.
I20260321 02:59:29.706892 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:29.749205 11235 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:29.729983 13424 raft_consensus.cc:3060] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:29.734715 13719 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
W20260321 02:59:29.756920 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 56741.4us
I20260321 02:59:29.755515 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:29.565511 13601 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:29.752523 11236 raft_consensus.cc:1240] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:29.766633 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:29.774866 13516 consensus_queue.cc:1059] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
W20260321 02:59:29.776435 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:29.787600 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:29.807844 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:29.813967 13549 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:29.818250 11232 raft_consensus.cc:1275] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 8 index: 3. (index mismatch)
I20260321 02:59:29.820677 13765 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:29.821383 13765 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:29.823657 13765 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:29.827618 13715 consensus_queue.cc:1059] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260321 02:59:29.829113 13600 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:29.987746 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:29.921025 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:29.921721 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:30.019424 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:29.873741 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:30.122584 11302 raft_consensus.cc:2368] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 7. Current term is 8.
I20260321 02:59:29.870187 11379 raft_consensus.cc:1240] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 4. Current term is 5. Ops: []
W20260321 02:59:30.124378 11349 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:30.126459 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:29.883337 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:29.976583 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:30.129421 11305 raft_consensus.cc:2368] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 6. Current term is 7.
I20260321 02:59:29.837790 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:30.094954 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:29.860831 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:30.119094 13799 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:30.133785 11376 raft_consensus.cc:1275] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260321 02:59:30.135942 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:30.138464 11133 catalog_manager.cc:5671] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 3 to 4, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4). New cstate: current_term: 4 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260321 02:59:30.140415 13781 log.cc:927] Time spent T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 2.199s user 0.013s sys 0.000s
I20260321 02:59:30.142150 11308 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1626) took 2899 ms. Trace:
I20260321 02:59:30.137050 11130 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42530 (request call id 219) took 10377 ms. Trace:
I20260321 02:59:30.142554 11308 rpcz_store.cc:276] 0321 02:59:27.242755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:27.243198 (+ 443us) service_pool.cc:225] Handling call
0321 02:59:27.289059 (+ 45861us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:27.556641 (+267582us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:27.556670 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:27.576674 (+ 20004us) log.cc:844] Serialized 26 byte log entry
0321 02:59:27.660848 (+ 84174us) raft_consensus.cc:1645] Marking committed up to 3
0321 02:59:27.660886 (+ 38us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:27.660956 (+ 70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:29.799593 (+2138637us) spinlock_profiling.cc:243] Waited 3.19 ms on lock 0x7b440002304c. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93c2dc5 00007fa8a93c37e2 00007fa8ac0f757b 00007fa8ac0f2e4f 00007fa8ac0e6c0e 00007fa8ac0e5f56 00007fa8ac0e5b78 00007fa8ac0ef5c1 00007fa8ac0ef579 00007fa8ac0ef509
0321 02:59:30.141664 (+342071us) raft_consensus.cc:1685] finished
0321 02:59:30.141690 (+ 26us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:30.141942 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12317,"raft.queue_time_us":18844,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":13649,"spinlock_wait_cycles":7329664,"thread_start_us":263804,"threads_started":2,"wal-append.queue_time_us":271951}
I20260321 02:59:30.145829 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:30.147327 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:30.138530 11310 raft_consensus.cc:2410] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260321 02:59:30.148103 11304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1653) took 1767 ms. Trace:
I20260321 02:59:30.148932 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:30.149144 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:29.974017 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:30.150570 11306 raft_consensus.cc:2393] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 8: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:30.155313 13549 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:30.101034 13505 raft_consensus.cc:3055] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:29.846171 13779 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:29.855808 13739 raft_consensus.cc:3055] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Stepping down as leader of term 7
W20260321 02:59:30.125031 11349 connection.cc:232] RPC call timeout handler was delayed by -1.6577585789999603s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18600000000000003s for a response
I20260321 02:59:30.103048 11200 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:30.102077 13594 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:29.853462 13845 raft_consensus.cc:493] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:29.975868 13513 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:30.155599 13567 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:30.148633 11304 rpcz_store.cc:276] 0321 02:59:28.360901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:28.417533 (+ 56632us) service_pool.cc:225] Handling call
0321 02:59:28.420060 (+ 2527us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:28.567091 (+147031us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:28.567120 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:29.271585 (+704465us) spinlock_profiling.cc:243] Waited 1.44 ms on lock 0x7b2c00010da8. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8aae48c81 00007fa8aae585e2 00007fa8ac587de1 00007fa8ac54c81a 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22
0321 02:59:29.301675 (+ 30090us) log.cc:844] Serialized 431 byte log entry
0321 02:59:29.362205 (+ 60530us) raft_consensus.cc:1645] Marking committed up to 4
0321 02:59:29.362256 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:30.115567 (+753311us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b54001accd4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:30.115639 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:30.115688 (+ 49us) raft_consensus.cc:1685] finished
0321 02:59:30.115702 (+ 14us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:30.128624 (+ 12922us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":340610560,"thread_start_us":60059,"threads_started":1,"wal-append.queue_time_us":232541}
I20260321 02:59:30.163350 11308 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:30.165477 13583 raft_consensus.cc:3055] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:29.864427 13564 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:30.168699 11376 raft_consensus.cc:1275] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260321 02:59:30.163378 13774 consensus_queue.cc:1059] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20260321 02:59:29.838245 13846 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:29.984287 13722 consensus_queue.cc:1048] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.423s
I20260321 02:59:30.419247 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:30.314363 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:30.421392 11385 raft_consensus.cc:2393] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 6: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:30.169044 13600 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:30.421500 11309 raft_consensus.cc:2393] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:30.144438 11130 rpcz_store.cc:276] 0321 02:59:19.771581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:19.771757 (+ 176us) service_pool.cc:225] Handling call
0321 02:59:30.136780 (+10365023us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:59:25.341808 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:59:25.501719 (+159911us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:59:25.519905 (+ 18186us) write_op.cc:435] Acquired schema lock
0321 02:59:25.528492 (+ 8587us) tablet.cc:661] Decoding operations
0321 02:59:25.713734 (+185242us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:59:25.739473 (+ 25739us) write_op.cc:641] Partition lock acquired for write op
0321 02:59:25.743507 (+ 4034us) tablet.cc:684] Acquiring locks for 8 operations
0321 02:59:25.826092 (+ 82585us) tablet.cc:700] Row locks acquired
0321 02:59:25.826138 (+ 46us) write_op.cc:260] PREPARE: finished
0321 02:59:25.937896 (+111758us) write_op.cc:270] Start()
0321 02:59:25.938096 (+ 200us) write_op.cc:276] Timestamp: P: 1774061965936654 usec, L: 0
0321 02:59:25.938142 (+ 46us) op_driver.cc:348] REPLICATION: starting
0321 02:59:25.939154 (+ 1012us) log.cc:844] Serialized 3190 byte log entry
0321 02:59:26.606353 (+667199us) op_driver.cc:464] REPLICATION: finished
0321 02:59:26.775281 (+168928us) write_op.cc:301] APPLY: starting
0321 02:59:26.778014 (+ 2733us) tablet.cc:1366] starting BulkCheckPresence
0321 02:59:26.783208 (+ 5194us) tablet.cc:1369] finished BulkCheckPresence
0321 02:59:26.798657 (+ 15449us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:59:26.809092 (+ 10435us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:59:26.809324 (+ 232us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0321 02:59:26.809388 (+ 64us) write_op.cc:312] APPLY: finished
0321 02:59:26.809665 (+ 277us) log.cc:844] Serialized 65 byte log entry
0321 02:59:26.809845 (+ 180us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:26.809966 (+ 121us) write_op.cc:454] Released schema lock
0321 02:59:26.810016 (+ 50us) write_op.cc:341] FINISH: Updating metrics
0321 02:59:26.810390 (+ 374us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:26.810428 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":503768,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":158827,"apply.run_cpu_time_us":3514,"apply.run_wall_time_us":42487,"num_ops":8,"prepare.queue_time_us":440643,"prepare.run_cpu_time_us":19765,"prepare.run_wall_time_us":968337,"raft.queue_time_us":38879,"raft.run_cpu_time_us":14987,"raft.run_wall_time_us":108012,"replication_time_us":668056,"thread_start_us":796519,"threads_started":4,"wal-append.queue_time_us":461932,"wal-append.run_cpu_time_us":23130,"wal-append.run_wall_time_us":1465083}]]}
W20260321 02:59:30.186592 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:30.378983 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:30.422032 13567 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:30.451615 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:30.183446 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:30.420819 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:30.244307 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:30.254026 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:30.420011 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:30.181299 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:30.453142 11307 raft_consensus.cc:2468] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:30.454770 11305 raft_consensus.cc:2410] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260321 02:59:30.453962 11304 raft_consensus.cc:2393] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 8: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:30.341780 11378 raft_consensus.cc:1240] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260321 02:59:30.331071 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:30.182554 11236 raft_consensus.cc:3055] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:30.274344 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:30.226011 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 123095us
I20260321 02:59:30.290701 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:30.465216 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:30.469007 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:30.420426 13717 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:30.469777 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:30.466287 13782 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.016s sys 0.019s
I20260321 02:59:30.466961 11302 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:30.289901 13715 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:30.468137 11382 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 4.
I20260321 02:59:30.244385 13779 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:30.167546 13505 raft_consensus.cc:740] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:30.466961 11236 raft_consensus.cc:740] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:30.467262 11303 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:30.481034 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:30.506986 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:30.506986 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:30.507054 11305 raft_consensus.cc:2393] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:30.480700 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:30.182749 13601 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:30.172287 13739 raft_consensus.cc:740] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:30.458813 13564 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:30.470239 11309 raft_consensus.cc:2393] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:30.458678 13583 raft_consensus.cc:740] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:30.334586 13553 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:30.226385 13593 raft_consensus.cc:2804] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:30.498514 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:30.473721 13718 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:30.421427 13845 raft_consensus.cc:515] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:30.508850 11235 raft_consensus.cc:2468] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:30.504182 13513 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:30.523918 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:30.513208 13719 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:30.531541 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:30.531805 11378 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:30.542660 11307 raft_consensus.cc:2393] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:30.532372 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:30.532130 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:30.528940 11304 raft_consensus.cc:1240] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260321 02:59:30.536064 11309 raft_consensus.cc:1240] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20260321 02:59:30.488339 13782 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.639s user 0.016s sys 0.019s
I20260321 02:59:30.530823 11236 consensus_queue.cc:260] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:30.549314 11236 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:30.539889 11230 raft_consensus.cc:2368] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 7. Current term is 8.
I20260321 02:59:30.550408 11310 raft_consensus.cc:2468] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 3.
W20260321 02:59:30.552662 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 136318us
I20260321 02:59:30.553956 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:30.557631 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:30.505963 13846 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:30.538285 13779 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:30.511858 13715 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:30.561975 11309 raft_consensus.cc:1240] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:30.560429 11975 raft_consensus.cc:1240] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:30.539456 13593 raft_consensus.cc:697] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:30.567806 11304 raft_consensus.cc:1275] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260321 02:59:30.531598 13553 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:30.575099 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:30.536178 13718 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:30.578987 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:30.579996 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:30.579154 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:30.574795 13854 log.cc:927] Time spent T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.281s user 0.000s sys 0.016s
I20260321 02:59:30.576416 13782 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:30.591009 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:30.588490 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:30.597245 11383 raft_consensus.cc:3060] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
W20260321 02:59:30.578027 13856 log.cc:927] Time spent T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.062s user 0.000s sys 0.001s
I20260321 02:59:30.597910 11385 raft_consensus.cc:2368] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7.
I20260321 02:59:30.593205 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:30.602356 11384 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:30.597164 11235 raft_consensus.cc:2410] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260321 02:59:30.598552 11382 raft_consensus.cc:2468] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:30.598740 11309 raft_consensus.cc:1275] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:30.600860 11379 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:30.596704 11380 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 6.
I20260321 02:59:30.605536 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:30.578995 13600 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:30.589708 13859 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:30.593092 13671 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:30.584164 13567 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:30.617334 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:30.618456 13553 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:30.621397 13845 leader_election.cc:290] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:30.625255 11310 raft_consensus.cc:1275] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260321 02:59:30.579566 13719 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:30.633635 13671 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:30.634583 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:30.636447 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:30.628149 11302 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:30.644367 11307 raft_consensus.cc:1240] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:30.682724 11236 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:30.686744 11385 raft_consensus.cc:2410] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260321 02:59:30.604573 13782 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:30.635844 13671 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:30.583273 13505 consensus_queue.cc:260] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:30.581341 13739 consensus_queue.cc:260] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:30.699494 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:30.700646 11235 raft_consensus.cc:2410] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260321 02:59:30.700802 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:30.701618 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 141317us
I20260321 02:59:30.591444 13583 consensus_queue.cc:260] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:30.705838 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:30.685528 13860 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:30.719929 11384 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:30.721782 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:30.722404 11385 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:30.723315 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:30.725442 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 161174us
I20260321 02:59:30.731783 11384 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
W20260321 02:59:30.733124 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:30.640760 13859 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:30.731850 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:30.744292 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:30.622388 13593 consensus_queue.cc:237] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:30.721447 13553 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:30.743572 13601 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:30.722110 13845 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:30.763532 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:30.712564 13600 consensus_queue.cc:1059] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260321 02:59:30.748701 13723 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:30.752009 13739 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:30.831880 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:30.827823 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:30.786617 11380 raft_consensus.cc:2393] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 6: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:30.835785 11380 raft_consensus.cc:1275] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260321 02:59:30.835930 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:30.836539 11376 raft_consensus.cc:1240] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260321 02:59:30.837322 11380 raft_consensus.cc:1240] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260321 02:59:30.838150 11376 raft_consensus.cc:1240] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:30.839125 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:30.749805 13513 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:30.842407 11231 raft_consensus.cc:2393] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:30.789739 13583 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:30.780833 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:30.832653 13715 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:30.743772 13860 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:30.781097 13845 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:30.883693 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:30.884410 11973 raft_consensus.cc:2468] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
W20260321 02:59:30.880324 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:30.762876 13553 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:30.882362 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:30.890895 11201 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:30.885329 13845 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:30.881497 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:30.771605 13601 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:30.887801 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:30.912006 11382 raft_consensus.cc:2410] T 515945e5685c4afe9d41d5317acb434c P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260321 02:59:30.877172 13779 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:30.875335 13846 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:30.917555 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:30.881807 13718 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:30.930119 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:30.931993 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:30.933072 13718 raft_consensus.cc:2804] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:30.933789 13718 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:30.934357 13718 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:30.939234 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:30.876274 13513 consensus_queue.cc:1048] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.041s
I20260321 02:59:30.940102 11980 raft_consensus.cc:2393] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 6: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
I20260321 02:59:30.944370 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:30.945279 13601 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:30.945709 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:30.946269 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:30.946846 11975 raft_consensus.cc:2393] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 6: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
W20260321 02:59:30.947288 13861 log.cc:927] Time spent T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.262s user 0.001s sys 0.000s
W20260321 02:59:30.933117 11944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:30.951211 11303 raft_consensus.cc:1275] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260321 02:59:30.952613 11303 raft_consensus.cc:1275] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:30.955935 13722 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:30.927445 11381 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45634 (request call id 343) took 1651 ms. Trace:
I20260321 02:59:30.957283 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:30.945868 13601 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:30.958887 11383 raft_consensus.cc:2468] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
W20260321 02:59:30.959378 11308 consensus_meta.cc:287] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.501s user 0.008s sys 0.002s
I20260321 02:59:30.839164 13723 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:30.971451 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:30.976720 11379 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:30.979609 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:30.967830 11384 raft_consensus.cc:1240] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260321 02:59:30.951869 13799 consensus_meta.cc:287] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.818s user 0.009s sys 0.000s
W20260321 02:59:30.957729 13867 log.cc:927] Time spent T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
W20260321 02:59:30.983922 11200 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:30.985368 11308 raft_consensus.cc:2410] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20260321 02:59:30.988884 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:30.991552 11235 raft_consensus.cc:2393] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:30.924206 13719 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:30.956813 11381 rpcz_store.cc:276] 0321 02:59:29.075192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:29.855445 (+780253us) service_pool.cc:225] Handling call
0321 02:59:29.856005 (+ 560us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:29.856546 (+ 541us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:29.856586 (+ 40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:29.857218 (+ 632us) log.cc:844] Serialized 26 byte log entry
0321 02:59:30.120809 (+263591us) raft_consensus.cc:1645] Marking committed up to 6
0321 02:59:30.120850 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:30.120918 (+ 68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:30.726621 (+605703us) raft_consensus.cc:1685] finished
0321 02:59:30.726640 (+ 19us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:30.726785 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1546,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":129,"thread_start_us":263136,"threads_started":1,"wal-append.queue_time_us":594665}
I20260321 02:59:30.987358 11275 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:30.995913 11303 raft_consensus.cc:1275] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260321 02:59:30.996503 11303 pending_rounds.cc:85] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da: Aborting all ops after (but not including) 3
I20260321 02:59:30.996799 11303 pending_rounds.cc:107] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260321 02:59:30.996934 11227 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:30.997135 11303 raft_consensus.cc:2889] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:30.998476 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:30.999224 11308 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:31.004182 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:30.930184 13859 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:31.000795 13863 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:31.008955 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:31.006486 13799 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:31.013093 13553 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:31.013556 13799 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:31.015626 13799 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:31.017598 11381 raft_consensus.cc:1240] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260321 02:59:31.019026 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:31.019943 11379 raft_consensus.cc:2410] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260321 02:59:31.019996 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:31.020907 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:31.021531 11380 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:31.022007 11231 raft_consensus.cc:3060] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:31.022336 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:31.023149 11380 raft_consensus.cc:2410] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260321 02:59:31.024793 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:31.025573 11381 raft_consensus.cc:2393] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 8: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:31.025849 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:31.026686 11381 raft_consensus.cc:1275] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:31.026752 11383 raft_consensus.cc:2393] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 7: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:31.027395 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:31.028014 11383 raft_consensus.cc:1275] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260321 02:59:31.028977 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:31.029124 11200 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:31.029866 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:31.030145 11385 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:31.030530 11235 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:31.030687 11380 raft_consensus.cc:1275] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260321 02:59:31.031064 11380 pending_rounds.cc:85] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5: Aborting all ops after (but not including) 3
I20260321 02:59:31.029955 11379 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:30.995743 13765 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:31.020068 11303 raft_consensus.cc:1275] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260321 02:59:31.032212 13765 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:31.032760 13733 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:31.033174 11303 raft_consensus.cc:1275] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:31.031311 11380 pending_rounds.cc:107] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260321 02:59:31.033746 11380 raft_consensus.cc:2889] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:31.034569 11200 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:31.035013 11380 raft_consensus.cc:1240] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:31.033253 13733 raft_consensus.cc:697] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:31.037007 13671 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:31.029670 11229 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 5.
I20260321 02:59:31.037577 13671 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:31.022887 13594 raft_consensus.cc:2955] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Committing config change with OpId 7.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } } }
I20260321 02:59:31.050853 11231 raft_consensus.cc:1275] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260321 02:59:31.050777 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:31.012197 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:31.037875 11381 raft_consensus.cc:1275] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260321 02:59:31.057945 11381 pending_rounds.cc:85] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5: Aborting all ops after (but not including) 2
I20260321 02:59:31.058331 11381 pending_rounds.cc:107] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260321 02:59:31.056339 11382 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:31.058684 11381 raft_consensus.cc:2889] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:31.038759 11380 raft_consensus.cc:1240] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260321 02:59:31.011482 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:31.061734 13782 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:31.037978 13733 consensus_queue.cc:237] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:31.062064 11305 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:31.062343 13782 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:31.062902 13782 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:31.064728 13873 raft_consensus.cc:2749] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260321 02:59:31.069528 13855 log.cc:927] Time spent T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.654s user 0.008s sys 0.004s
W20260321 02:59:31.071518 13857 log.cc:927] Time spent T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.497s user 0.006s sys 0.012s
I20260321 02:59:31.074376 11385 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:31.059736 11309 raft_consensus.cc:2468] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:31.075263 11383 raft_consensus.cc:1240] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:31.075547 11228 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:30.956496 13722 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:31.082496 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:30.986589 13601 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:31.083451 11379 raft_consensus.cc:1275] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260321 02:59:31.087037 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:31.086726 11379 pending_rounds.cc:85] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5: Aborting all ops after (but not including) 3
I20260321 02:59:31.092590 11379 pending_rounds.cc:107] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260321 02:59:31.092980 11384 raft_consensus.cc:1275] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260321 02:59:31.093075 11379 raft_consensus.cc:2889] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:31.074095 13799 raft_consensus.cc:2804] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260321 02:59:31.092006 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 144040us
I20260321 02:59:31.096462 11236 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58214 (request call id 1322) took 1330 ms. Trace:
I20260321 02:59:31.096660 13750 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:31.097935 11380 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:31.105202 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:31.102262 11227 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58214 (request call id 1317) took 1559 ms. Trace:
I20260321 02:59:31.104812 11236 rpcz_store.cc:276] 0321 02:59:29.765999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:29.772228 (+ 6229us) service_pool.cc:225] Handling call
0321 02:59:31.096270 (+1324042us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b5400061154. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:31.096395 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":891081600}
I20260321 02:59:31.107293 11227 rpcz_store.cc:276] 0321 02:59:29.542644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:29.561733 (+ 19089us) service_pool.cc:225] Handling call
0321 02:59:31.102078 (+1540345us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b5400264654. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:31.102192 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3147771648}
I20260321 02:59:31.093677 11384 pending_rounds.cc:85] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5: Aborting all ops after (but not including) 3
I20260321 02:59:31.108712 11384 pending_rounds.cc:107] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260321 02:59:31.108829 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:31.083639 13717 consensus_queue.cc:1048] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20260321 02:59:31.106003 11376 raft_consensus.cc:2376] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8. Re-sending same reply.
I20260321 02:59:31.109005 11384 raft_consensus.cc:2889] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:31.120442 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:31.116513 11379 raft_consensus.cc:2393] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 5: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
W20260321 02:59:31.122481 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:31.121634 11384 raft_consensus.cc:3055] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:31.125095 11376 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:31.125722 11384 raft_consensus.cc:740] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:31.011822 13863 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:31.129266 11234 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:31.107664 13799 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:31.145973 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:31.146267 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:31.147091 11376 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:31.122992 13860 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:31.119201 13671 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:31.153179 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:31.159641 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:31.133800 13601 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:31.160773 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:31.160827 11383 raft_consensus.cc:2393] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 8: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:31.114462 13750 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:31.144672 13799 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
W20260321 02:59:31.166946 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 446624us
I20260321 02:59:31.140936 11384 consensus_queue.cc:260] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:31.167925 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 277238us
W20260321 02:59:31.203728 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 166830us
W20260321 02:59:31.210549 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:31.212237 11384 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:31.176132 11349 leader_election.cc:400] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 6. Current term is 7.
I20260321 02:59:31.218756 11232 raft_consensus.cc:2410] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20260321 02:59:31.220281 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:31.221503 13717 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:31.205993 13723 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:31.200940 13782 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:31.216856 13564 consensus_meta.cc:287] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.686s user 0.008s sys 0.004s
I20260321 02:59:31.236284 13564 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:31.237802 11234 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58214 (request call id 1312) took 1696 ms (client timeout 1729 ms). Trace:
I20260321 02:59:31.238854 13564 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:31.238870 11234 rpcz_store.cc:269] 0321 02:59:29.540988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:29.581695 (+ 40707us) service_pool.cc:225] Handling call
0321 02:59:31.237466 (+1655771us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b5400289bd4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:31.237670 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3404986112}
I20260321 02:59:31.241355 11310 raft_consensus.cc:1240] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:31.227208 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:31.200249 13860 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:31.221577 13880 consensus_queue.cc:1059] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20260321 02:59:31.230031 13505 raft_consensus.cc:3060] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:31.243371 11980 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:31.229887 11275 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:31.221096 11942 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:31.233548 11384 raft_consensus.cc:1275] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 3. (term mismatch)
I20260321 02:59:31.250728 11384 pending_rounds.cc:85] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5: Aborting all ops after (but not including) 2
I20260321 02:59:31.251188 11384 pending_rounds.cc:107] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260321 02:59:31.251724 11384 raft_consensus.cc:2889] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260321 02:59:31.252319 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:31.249630 11201 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:31.253878 11384 pending_rounds.cc:107] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260321 02:59:31.254609 11384 raft_consensus.cc:2889] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:31.246510 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:31.257839 13882 log.cc:927] Time spent T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
W20260321 02:59:31.259698 13853 log.cc:927] Time spent T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.895s user 0.008s sys 0.008s
I20260321 02:59:31.261854 13860 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:31.264142 13860 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:31.265039 13722 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:31.266744 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 107469us
I20260321 02:59:31.268501 11349 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:31.269608 13779 raft_consensus.cc:3055] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:31.270007 13779 raft_consensus.cc:740] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:31.270541 13782 raft_consensus.cc:2749] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:31.270750 11349 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:31.270954 13779 consensus_queue.cc:260] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:31.224934 13593 consensus_queue.cc:1048] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.026s
W20260321 02:59:31.301210 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 51707.6us
I20260321 02:59:31.303910 13886 raft_consensus.cc:2804] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:31.304548 13886 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:31.305307 13886 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:31.306800 13779 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:31.284552 13887 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:31.308676 13887 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:31.312819 11306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49786 (request call id 216) took 1166 ms. Trace:
I20260321 02:59:31.313237 11306 rpcz_store.cc:276] 0321 02:59:30.146816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:30.472801 (+325985us) service_pool.cc:225] Handling call
0321 02:59:30.473133 (+ 332us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:30.701425 (+228292us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:30.701489 (+ 64us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:30.751045 (+ 49556us) log.cc:844] Serialized 26 byte log entry
0321 02:59:30.992466 (+241421us) raft_consensus.cc:1645] Marking committed up to 6
0321 02:59:30.992507 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:30.992592 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:31.312604 (+320012us) raft_consensus.cc:1685] finished
0321 02:59:31.312626 (+ 22us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:31.312757 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":144967,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":43087,"thread_start_us":405354,"threads_started":2,"wal-append.queue_time_us":439610}
I20260321 02:59:31.318579 13583 raft_consensus.cc:2749] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260321 02:59:31.321429 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 50743.1us
W20260321 02:59:31.323038 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:31.322620 11349 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:31.328567 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 107637us
I20260321 02:59:31.284461 13600 raft_consensus.cc:2749] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260321 02:59:31.330940 11228 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58214 (request call id 1313) took 1789 ms (client timeout 1674 ms). Trace:
W20260321 02:59:31.338543 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 109101us
I20260321 02:59:31.338544 11942 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:31.311642 13887 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:31.255669 13718 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:31.228816 13717 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:31.348815 11380 raft_consensus.cc:1275] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:31.301064 13889 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:31.274860 13765 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:31.297462 13567 raft_consensus.cc:2804] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election won for term 4
W20260321 02:59:31.349498 11228 rpcz_store.cc:269] 0321 02:59:29.541331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:29.636563 (+ 95232us) service_pool.cc:225] Handling call
0321 02:59:31.330695 (+1694132us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b540037ad54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:31.330847 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2724423040}
W20260321 02:59:31.354377 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 68348.4us
W20260321 02:59:31.302749 13881 log.cc:927] Time spent T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.088s user 0.000s sys 0.004s
I20260321 02:59:31.363997 13664 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:31.364189 13893 raft_consensus.cc:2749] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:31.364799 13664 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:31.358021 13719 consensus_queue.cc:1048] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260321 02:59:31.366997 13664 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:31.372138 13877 raft_consensus.cc:2804] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:31.372485 13893 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:31.372704 13877 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:31.373095 13877 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:31.373027 13893 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:31.373238 13671 raft_consensus.cc:3055] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:31.374949 11349 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:31.348834 13739 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:31.381956 11311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56616 (request call id 1703) took 1262 ms. Trace:
I20260321 02:59:31.327997 13723 consensus_queue.cc:1048] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.056s
I20260321 02:59:31.404135 11311 rpcz_store.cc:276] 0321 02:59:30.123633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:30.472207 (+348574us) service_pool.cc:225] Handling call
0321 02:59:30.507925 (+ 35718us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:30.860044 (+352119us) raft_consensus.cc:1534] Early marking committed up to index 2
0321 02:59:30.860088 (+ 44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:30.881050 (+ 20962us) log.cc:844] Serialized 26 byte log entry
0321 02:59:31.196199 (+315149us) raft_consensus.cc:1645] Marking committed up to 2
0321 02:59:31.196242 (+ 43us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:31.196319 (+ 77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:31.307819 (+111500us) raft_consensus.cc:1685] finished
0321 02:59:31.307853 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:31.362849 (+ 54996us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x7b5400319f50. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54bc8d 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:31.381889 (+ 19040us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":169700,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":5537,"spinlock_wait_cycles":38332288,"thread_start_us":595460,"threads_started":2,"wal-append.queue_time_us":396352}
I20260321 02:59:31.353559 13888 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:31.385278 11200 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
W20260321 02:59:31.428025 11944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:31.438867 11349 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:31.353024 13795 consensus_queue.cc:1048] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
I20260321 02:59:31.435915 13865 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:31.442281 13860 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:31.442517 13779 raft_consensus.cc:2749] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:31.403888 13600 raft_consensus.cc:2749] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:31.460096 13865 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:31.460227 13860 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:31.437749 13564 consensus_queue.cc:1059] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.005s
I20260321 02:59:31.461553 13564 raft_consensus.cc:3055] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:31.462065 13564 raft_consensus.cc:740] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:31.462378 13865 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:31.462920 13564 consensus_queue.cc:260] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:31.463619 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:31.464442 13564 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:31.462409 13860 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:31.443625 11349 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:31.445540 11275 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:31.468888 13860 raft_consensus.cc:2764] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260321 02:59:31.468976 11275 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:31.402859 13739 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:31.469892 13860 raft_consensus.cc:2804] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:31.470005 11275 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:31.470302 13860 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:31.470570 11349 leader_election.cc:400] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 7. Current term is 8.
I20260321 02:59:31.470937 11349 leader_election.cc:403] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260321 02:59:31.456194 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:31.456357 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 71198.4us
W20260321 02:59:31.457249 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:31.457759 13717 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:31.459250 11351 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:31.470834 13865 raft_consensus.cc:2749] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:31.471160 13739 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:31.472019 13600 raft_consensus.cc:2749] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260321 02:59:31.472362 13895 consensus_queue.cc:1048] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:59:31.470718 13860 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:31.383100 13671 raft_consensus.cc:740] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:31.455303 13723 raft_consensus.cc:2804] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260321 02:59:31.437860 11942 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:31.500639 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:31.490470 13739 consensus_queue.cc:1048] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260321 02:59:31.502288 11383 raft_consensus.cc:1240] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:31.501813 13733 consensus_queue.cc:1048] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:59:31.505285 11200 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:31.490181 13600 raft_consensus.cc:2749] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 7. Current term is 8.
I20260321 02:59:31.413625 13567 raft_consensus.cc:697] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 4 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:31.479137 13863 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:31.455602 13886 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:31.491187 13723 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:31.361531 13889 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:31.530644 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:31.548211 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:31.550315 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:31.530210 13723 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:31.540513 13863 raft_consensus.cc:2804] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260321 02:59:31.575045 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 44879.5us
W20260321 02:59:31.577346 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:31.541028 13750 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:31.563705 13671 consensus_queue.cc:260] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:31.584237 13671 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:31.584569 13865 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:31.585319 13865 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:31.491659 13888 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:31.602224 13901 log.cc:927] Time spent T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20260321 02:59:31.578203 13863 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:31.577678 13594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:59200 (request call id 12) took 5735 ms. Trace:
I20260321 02:59:31.693029 11349 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
W20260321 02:59:31.695480 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 201369us
I20260321 02:59:31.640589 13718 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:31.708245 11975 raft_consensus.cc:1275] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260321 02:59:31.708652 11975 pending_rounds.cc:85] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56: Aborting all ops after (but not including) 6
I20260321 02:59:31.708873 11975 pending_rounds.cc:107] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56: Aborting uncommitted NO_OP operation due to leader change: 6.7
W20260321 02:59:31.685345 13905 log.cc:927] Time spent T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
W20260321 02:59:31.684607 13900 log.cc:927] Time spent T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.132s user 0.000s sys 0.001s
W20260321 02:59:31.714030 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 113349us
I20260321 02:59:31.714409 11975 raft_consensus.cc:2889] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:31.716501 11275 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:31.717206 11942 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.762s (SENT) [suppressed 24 similar messages]
I20260321 02:59:31.717391 11275 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:31.607076 13865 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:31.742199 11306 raft_consensus.cc:2955] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Committing config change with OpId 7.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } } }
I20260321 02:59:31.743070 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:31.744285 13873 raft_consensus.cc:2804] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:31.744872 13873 raft_consensus.cc:697] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:31.745759 13873 consensus_queue.cc:237] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:31.745971 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:31.745919 13877 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:31.746802 11304 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:31.747802 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:31.741317 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:31.748575 11302 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:31.748795 11383 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:31.748981 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:31.674316 13893 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:31.749428 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:31.719568 13846 consensus_queue.cc:1048] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.035s
I20260321 02:59:31.748123 13877 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:31.751219 11377 raft_consensus.cc:1275] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:31.753919 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:31.722088 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:31.756822 11311 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:31.775220 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:31.776187 11378 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:31.718328 11275 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:31.696467 13863 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:31.706697 13782 raft_consensus.cc:2749] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:31.649469 13799 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:31.780519 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 63332us
I20260321 02:59:31.738569 13739 consensus_queue.cc:1059] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 1, Time since last communication: 0.001s
I20260321 02:59:31.739375 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:31.740447 11385 raft_consensus.cc:2955] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Committing config change with OpId 7.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } } }
I20260321 02:59:31.741286 13889 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:31.741556 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:31.741740 13564 raft_consensus.cc:2804] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:31.696669 13594 rpcz_store.cc:276] 0321 02:59:25.814945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:25.815399 (+ 454us) service_pool.cc:225] Handling call
0321 02:59:26.030981 (+215582us) log.cc:844] Serialized 431 byte log entry
0321 02:59:31.544249 (+5513268us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":71852,"raft.queue_time_us":174797,"raft.run_cpu_time_us":906,"raft.run_wall_time_us":69942,"thread_start_us":447824,"threads_started":2,"wal-append.queue_time_us":416853,"wal-append.run_cpu_time_us":2012,"wal-append.run_wall_time_us":1085975}
I20260321 02:59:31.785496 11379 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
W20260321 02:59:31.772791 13904 log.cc:927] Time spent T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.173s user 0.000s sys 0.001s
I20260321 02:59:31.790165 11942 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:31.792649 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:31.788460 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:31.793411 11942 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
W20260321 02:59:31.798683 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:31.788515 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:31.790774 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:31.715657 13886 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:31.793231 11381 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:31.802971 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:31.793740 11303 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:31.804764 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:31.805693 11304 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
W20260321 02:59:31.806376 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:31.805495 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 87229.8us
I20260321 02:59:31.808014 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:31.808748 11310 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:31.816540 11275 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:31.819432 11975 raft_consensus.cc:1240] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:31.652619 13567 consensus_queue.cc:237] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:31.824963 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:31.826781 11303 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:31.788196 13564 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:31.827663 11302 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
W20260321 02:59:31.832546 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:31.809698 13671 raft_consensus.cc:2804] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:31.810943 13865 raft_consensus.cc:2804] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:31.809980 13870 raft_consensus.cc:2804] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:31.838991 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:31.843215 11980 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Advancing to term 6
W20260321 02:59:31.843550 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:31.811695 13799 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:31.870554 11944 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:31.880559 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 104944us
I20260321 02:59:31.812031 13715 consensus_queue.cc:1048] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.037s
I20260321 02:59:31.840014 13564 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
W20260321 02:59:31.890089 11942 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:31.895428 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:31.904613 11228 raft_consensus.cc:2393] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:31.897233 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:31.897538 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:31.906550 11236 raft_consensus.cc:2468] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 4.
I20260321 02:59:31.898392 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:31.842778 13671 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:31.908735 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:31.911237 11310 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:31.912814 11200 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:31.848182 13739 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:31.848168 13870 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:31.923074 11200 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:31.929891 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:31.931533 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:31.882416 13846 raft_consensus.cc:2804] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:31.936352 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:31.886015 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
W20260321 02:59:31.939656 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:31.940158 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 131958us
I20260321 02:59:31.940241 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:31.942183 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:31.880095 13865 raft_consensus.cc:697] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:31.948197 11275 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:31.948781 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:31.950188 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:31.892983 13719 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:31.944854 11228 raft_consensus.cc:1240] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:31.909065 13671 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Advancing to term 5
W20260321 02:59:31.954137 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:31.857393 13750 consensus_queue.cc:1048] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.080s
I20260321 02:59:31.929548 13799 raft_consensus.cc:2749] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:31.938407 13718 raft_consensus.cc:2749] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260321 02:59:31.949115 13907 log.cc:927] Time spent T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.297s user 0.001s sys 0.001s
W20260321 02:59:31.960114 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 93995us
I20260321 02:59:31.879437 13895 consensus_queue.cc:1059] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.066s
I20260321 02:59:31.933781 13870 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Advancing to term 7
W20260321 02:59:31.960632 11940 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:31.946249 13888 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:31.999822 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 55064.4us
I20260321 02:59:31.937547 13739 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:32.009409 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:31.971999 13733 raft_consensus.cc:3055] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Stepping down as leader of term 8
I20260321 02:59:32.016893 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:32.019619 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:32.020548 11235 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:32.021762 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:32.023468 11275 connection.cc:232] RPC call timeout handler was delayed by -2.178893592999998s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1593s for a response
I20260321 02:59:32.025458 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:32.024821 11275 connection.cc:232] RPC call timeout handler was delayed by -2.0981330450000542s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.163s for a response
W20260321 02:59:32.034000 11275 connection.cc:232] RPC call timeout handler was delayed by -2.065493593000042s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17190000000000003s for a response
W20260321 02:59:32.034423 11275 connection.cc:232] RPC call timeout handler was delayed by -2.0231935930000873s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1766s for a response
W20260321 02:59:32.034788 11275 connection.cc:232] RPC call timeout handler was delayed by -1.8863935930000935s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1918s for a response
I20260321 02:59:32.022814 11235 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:32.035924 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:32.032503 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:32.036401 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:32.036800 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:31.955663 13719 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:32.042434 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 119484us
W20260321 02:59:32.039819 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:32.006971 13846 raft_consensus.cc:697] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:32.041275 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:32.041481 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:32.060111 11200 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:32.014904 13873 raft_consensus.cc:2804] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:32.063443 11200 leader_election.cc:304] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:32.013978 13733 raft_consensus.cc:740] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
W20260321 02:59:32.076160 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 102762us
W20260321 02:59:32.084077 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:32.052137 13846 consensus_queue.cc:237] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:32.081554 11233 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:32.088631 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:32.090358 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:32.090943 11378 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:32.091768 11349 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:32.091881 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:32.094198 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:32.094535 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:32.094846 11379 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:32.095103 11382 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:32.095702 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:32.096275 11379 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:32.097154 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:32.092839 11378 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:32.097828 13873 raft_consensus.cc:697] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:32.107116 11200 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:32.065634 13880 consensus_queue.cc:1048] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.042s
I20260321 02:59:32.104933 13873 consensus_queue.cc:237] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:32.108165 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:32.109405 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:32.109979 13799 raft_consensus.cc:2749] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:32.110278 11376 raft_consensus.cc:2393] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 8: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:32.102802 11980 raft_consensus.cc:2468] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:32.014407 13723 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:32.069372 13895 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:32.121852 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:32.126420 11201 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:32.129138 11380 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:32.126333 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:32.131225 11381 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:32.068682 13924 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
W20260321 02:59:32.100680 13860 consensus_meta.cc:287] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.627s user 0.013s sys 0.000s
I20260321 02:59:32.083547 13782 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:32.069301 13664 consensus_queue.cc:1048] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.071s
I20260321 02:59:32.098778 13594 raft_consensus.cc:2749] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:32.146739 11229 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:32.120173 13765 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:32.141804 13779 raft_consensus.cc:2749] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260321 02:59:32.167634 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 75976.8us
W20260321 02:59:32.171823 11275 connection.cc:232] RPC call timeout handler was delayed by -1.8026935930000718s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20110000000000003s for a response
W20260321 02:59:32.172407 11275 connection.cc:232] RPC call timeout handler was delayed by -1.7657935930000122s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20520000000000002s for a response
W20260321 02:59:32.172832 11275 connection.cc:232] RPC call timeout handler was delayed by -1.7474615150000545s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1502s for a response
W20260321 02:59:32.173210 11275 connection.cc:232] RPC call timeout handler was delayed by -1.6136935929999936s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22210000000000002s for a response
W20260321 02:59:32.173661 11275 connection.cc:232] RPC call timeout handler was delayed by -1.5926615150000316s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1674s for a response
W20260321 02:59:32.174124 11275 connection.cc:232] RPC call timeout handler was delayed by -1.5431615150000653s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17290000000000003s for a response
W20260321 02:59:32.174556 11275 connection.cc:232] RPC call timeout handler was delayed by -1.4234615150000991s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18620000000000003s for a response
W20260321 02:59:32.174928 11275 connection.cc:232] RPC call timeout handler was delayed by -1.3595615150001095s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19330000000000003s for a response
W20260321 02:59:32.175310 11275 connection.cc:232] RPC call timeout handler was delayed by -1.2565002830000367s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16200000000000003s for a response
W20260321 02:59:32.112744 11182 auto_rebalancer.cc:799] Could not move replica: Incomplete: tablet 513757b127054285b0686cd45044c596, TS fef7f8cc55b043039131bd9b2229f0a5 -> TS 3d8d3176d6ec407ea2a965556c2e2c56 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:32.178284 11182 auto_rebalancer.cc:284] scheduled replica move failed to complete: Incomplete: tablet 513757b127054285b0686cd45044c596, TS fef7f8cc55b043039131bd9b2229f0a5 -> TS 3d8d3176d6ec407ea2a965556c2e2c56 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:32.175760 11275 connection.cc:232] RPC call timeout handler was delayed by -1.1960935930000005s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2685s for a response
W20260321 02:59:32.180554 11275 connection.cc:232] RPC call timeout handler was delayed by -1.1098002830000269s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17830000000000001s for a response
W20260321 02:59:32.181162 11275 connection.cc:232] RPC call timeout handler was delayed by -1.076366541000084s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1603s for a response
W20260321 02:59:32.181514 11275 connection.cc:232] RPC call timeout handler was delayed by -1.0520665410000447s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.163s for a response
I20260321 02:59:32.183048 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:32.146238 13920 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:32.195294 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:32.135569 13718 consensus_queue.cc:1048] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260321 02:59:32.082563 13865 consensus_queue.cc:237] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:32.198904 11275 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:32.141945 13750 consensus_queue.cc:1048] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.041s
I20260321 02:59:32.164515 13888 raft_consensus.cc:2804] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:32.216747 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:32.226502 11310 raft_consensus.cc:2468] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:32.227433 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:32.205843 13930 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:32.147323 13860 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:32.150240 13782 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:32.236195 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:32.149762 13924 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:32.168134 13765 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:32.237536 13921 log.cc:927] Time spent T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.281s user 0.001s sys 0.000s
W20260321 02:59:32.249914 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 139558us
I20260321 02:59:32.256300 13933 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:32.248601 13929 consensus_queue.cc:1059] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.107s
I20260321 02:59:32.260241 13799 raft_consensus.cc:3055] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:32.260603 13799 raft_consensus.cc:740] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:32.260675 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:32.261431 13799 consensus_queue.cc:260] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:32.262738 13799 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:32.264479 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:32.266512 11310 raft_consensus.cc:2468] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:59:32.233659 13888 raft_consensus.cc:697] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:32.217188 13920 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:32.318514 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:32.355837 11275 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
W20260321 02:59:32.356068 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 76916.3us
I20260321 02:59:32.355923 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:32.362560 11310 raft_consensus.cc:2393] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 9: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:32.194780 13733 consensus_queue.cc:260] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:32.356477 11942 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:32.373631 13893 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:32.374259 13719 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:32.376238 13924 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:32.377053 13916 log.cc:927] Time spent T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.281s user 0.003s sys 0.001s
I20260321 02:59:32.381137 13860 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:32.382503 13723 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:32.388031 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:32.389451 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:32.390105 11311 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 8.
I20260321 02:59:32.388070 13779 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:32.392669 13779 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:32.393304 13765 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:32.393781 13782 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:32.395032 13779 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:32.398813 11304 raft_consensus.cc:2468] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:59:32.399116 13920 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:32.400020 13888 consensus_queue.cc:237] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:32.400738 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 45096.1us
I20260321 02:59:32.401898 11275 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
W20260321 02:59:32.373901 13863 consensus_meta.cc:287] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.595s user 0.010s sys 0.000s
W20260321 02:59:32.407830 11385 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.006s sys 0.024s
W20260321 02:59:32.408335 11385 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.590s user 0.006s sys 0.024s
I20260321 02:59:32.359163 13739 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:32.236758 13930 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:32.415068 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:32.411594 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:32.364367 13593 raft_consensus.cc:3055] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:32.425927 11311 raft_consensus.cc:1275] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260321 02:59:32.363289 13594 consensus_queue.cc:1048] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
W20260321 02:59:32.477814 11303 consensus_meta.cc:287] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.632s user 0.009s sys 0.001s
I20260321 02:59:32.410698 13938 raft_consensus.cc:2804] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260321 02:59:32.480790 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:32.516140 11303 raft_consensus.cc:2468] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:32.517767 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:32.418334 13919 log.cc:927] Time spent T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.151s user 0.000s sys 0.000s
W20260321 02:59:32.522897 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:32.415897 13733 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:32.363895 13717 consensus_queue.cc:1048] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.134s
W20260321 02:59:32.512341 13564 consensus_meta.cc:287] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.556s user 0.016s sys 0.005s
W20260321 02:59:32.415529 13921 log.cc:927] Time spent T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.136s user 0.000s sys 0.000s
I20260321 02:59:32.532737 13933 raft_consensus.cc:2749] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:32.537375 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:32.515834 13593 raft_consensus.cc:740] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:32.523236 13924 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:32.374274 13893 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:32.410527 13863 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:32.415611 13750 consensus_queue.cc:1048] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.113s
I20260321 02:59:32.416406 13860 raft_consensus.cc:2804] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260321 02:59:32.546598 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:32.516067 13938 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:32.476967 13719 consensus_queue.cc:1048] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
W20260321 02:59:32.539050 13870 consensus_meta.cc:287] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 0.539s user 0.010s sys 0.003s
I20260321 02:59:32.530769 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:32.551230 11384 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45634 (request call id 357) took 1055 ms. Trace:
W20260321 02:59:32.491641 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 81649.1us
I20260321 02:59:32.545423 13564 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:32.631217 11182 auto_rebalancer.cc:791] Removing replace marker failed: Replica fef7f8cc55b043039131bd9b2229f0a5 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
W20260321 02:59:32.640404 11182 auto_rebalancer.cc:799] Could not move replica: Incomplete: tablet c104d8b810534a6c918eab2f919a83dc, TS 0d7efc590a1041a9bd4cd4a971332dfb -> TS 3d8d3176d6ec407ea2a965556c2e2c56 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:32.640764 11182 auto_rebalancer.cc:284] scheduled replica move failed to complete: Incomplete: tablet c104d8b810534a6c918eab2f919a83dc, TS 0d7efc590a1041a9bd4cd4a971332dfb -> TS 3d8d3176d6ec407ea2a965556c2e2c56 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:32.636891 11384 rpcz_store.cc:276] 0321 02:59:31.495571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:31.495745 (+ 174us) service_pool.cc:225] Handling call
0321 02:59:31.496211 (+ 466us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:31.496664 (+ 453us) raft_consensus.cc:1534] Early marking committed up to index 7
0321 02:59:31.695919 (+199255us) log.cc:844] Serialized 14 byte log entry
0321 02:59:31.808605 (+112686us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:31.818912 (+ 10307us) log.cc:844] Serialized 26 byte log entry
0321 02:59:31.819007 (+ 95us) raft_consensus.cc:1645] Marking committed up to 7
0321 02:59:31.819029 (+ 22us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:31.819114 (+ 85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:32.550964 (+731850us) raft_consensus.cc:1685] finished
0321 02:59:32.550993 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:32.551154 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":51257,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":183,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":13261,"spinlock_wait_cycles":68352,"thread_start_us":248925,"threads_started":2,"wal-append.queue_time_us":169061}
I20260321 02:59:32.642372 11236 raft_consensus.cc:2393] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 9: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:32.629760 11310 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
W20260321 02:59:32.557808 13671 consensus_meta.cc:287] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 0.589s user 0.011s sys 0.000s
I20260321 02:59:32.639279 11200 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:32.647153 11200 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:32.650022 11200 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:32.653122 11200 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:32.624917 13860 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:32.630741 13938 raft_consensus.cc:3060] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:32.657017 11200 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:32.654862 13664 raft_consensus.cc:2749] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:32.669296 13893 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:32.670960 13930 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:32.672482 11980 raft_consensus.cc:1275] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260321 02:59:32.614893 13924 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:32.656526 13939 consensus_queue.cc:1048] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260321 02:59:32.674925 13664 raft_consensus.cc:2804] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:32.648921 13594 raft_consensus.cc:2804] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:32.662899 13860 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:32.665524 13718 raft_consensus.cc:2804] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:32.642062 13886 consensus_queue.cc:1048] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260321 02:59:32.697220 11980 raft_consensus.cc:1240] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:32.687511 13920 consensus_queue.cc:1048] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260321 02:59:32.640036 13870 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:32.661389 13739 raft_consensus.cc:2804] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:32.649194 13671 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:32.683914 13594 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:32.681552 13664 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:32.697338 13718 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:32.712244 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 310474us
I20260321 02:59:32.714273 11275 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:32.700472 13739 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:32.704602 13594 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:32.666261 13593 consensus_queue.cc:260] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:32.704187 13664 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:32.726416 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:32.728814 11974 raft_consensus.cc:2393] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 5: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
I20260321 02:59:32.728981 13750 consensus_queue.cc:1059] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260321 02:59:32.726127 11973 raft_consensus.cc:2368] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7.
I20260321 02:59:32.729069 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:32.731194 13671 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:32.720588 13718 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:32.732734 13924 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:32.744442 13944 log.cc:927] Time spent T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
I20260321 02:59:32.744793 13664 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:32.746628 13870 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:32.751483 13948 log.cc:927] Time spent T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
W20260321 02:59:32.752384 13928 log.cc:927] Time spent T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.594s user 0.003s sys 0.000s
I20260321 02:59:32.708348 13863 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:32.739705 13895 consensus_meta.cc:287] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.602s user 0.008s sys 0.000s
I20260321 02:59:32.721613 13739 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:32.770332 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:32.771454 11275 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:32.720240 13846 raft_consensus.cc:2749] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:32.724797 13719 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:32.779855 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:32.782704 11349 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:32.791555 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:32.792521 11980 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
W20260321 02:59:32.795573 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:32.801077 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:32.802064 11384 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:32.802661 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:32.803292 11379 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:32.804745 11310 raft_consensus.cc:1275] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260321 02:59:32.807384 11303 raft_consensus.cc:1240] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:32.814667 11385 raft_consensus.cc:1275] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260321 02:59:32.814697 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:32.816593 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:32.819759 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:32.820545 11384 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:32.815769 11377 raft_consensus.cc:2376] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5. Re-sending same reply.
W20260321 02:59:32.827018 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 65680.3us
I20260321 02:59:32.836360 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:32.844290 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:32.849980 11303 raft_consensus.cc:1275] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:32.773871 13895 raft_consensus.cc:2749] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:32.775679 13924 raft_consensus.cc:2804] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:32.839429 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:32.854050 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:32.860216 11302 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:32.793484 13593 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:32.814291 13779 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:32.817210 13893 raft_consensus.cc:2804] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:32.880836 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:32.885046 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:32.827008 13889 consensus_queue.cc:1059] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.057s
I20260321 02:59:32.903687 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:32.817602 13719 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:32.875218 13924 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:32.934187 11275 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:32.919548 11310 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:32.878556 13564 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:32.904673 11303 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:32.922653 11276 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:32.956650 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:32.957592 11311 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:32.945436 13924 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:32.884028 13893 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
W20260321 02:59:32.949394 11942 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:32.890075 13779 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:32.947363 13889 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:32.963291 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:32.974779 11378 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:32.941406 11302 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
W20260321 02:59:32.973402 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 61048.7us
I20260321 02:59:32.977360 11974 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45078 (request call id 487) took 1153 ms. Trace:
I20260321 02:59:32.977917 11974 rpcz_store.cc:276] 0321 02:59:31.818606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:31.820312 (+ 1706us) service_pool.cc:225] Handling call
0321 02:59:32.960342 (+1140030us) spinlock_profiling.cc:243] Waited 765 ms on lock 0x7b540063e554. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:32.970784 (+ 10442us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1758482176}
I20260321 02:59:32.954773 13938 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:32.980834 13719 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:32.981071 13938 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:32.984162 13664 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:32.908195 13923 consensus_queue.cc:1059] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20260321 02:59:32.986526 13923 raft_consensus.cc:3055] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:32.986967 13923 raft_consensus.cc:740] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
W20260321 02:59:32.992651 13925 log.cc:927] Time spent T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.908s user 0.001s sys 0.003s
I20260321 02:59:32.970888 13893 raft_consensus.cc:3060] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:32.955992 13920 raft_consensus.cc:2749] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260321 02:59:33.000844 13889 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.020241 11305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1701) took 1289 ms. Trace:
I20260321 02:59:33.020851 11305 rpcz_store.cc:276] 0321 02:59:31.730714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:31.731011 (+ 297us) service_pool.cc:225] Handling call
0321 02:59:31.731480 (+ 469us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:31.846221 (+114741us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:31.846250 (+ 29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:31.846723 (+ 473us) log.cc:844] Serialized 26 byte log entry
0321 02:59:31.893407 (+ 46684us) raft_consensus.cc:1645] Marking committed up to 3
0321 02:59:31.893448 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:31.893552 (+ 104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:32.946717 (+1053165us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b440002304c. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93c2dc5 00007fa8a93c37e2 00007fa8ac0f757b 00007fa8ac0f2e4f 00007fa8ac0e6c0e 00007fa8ac0e5f56 00007fa8ac0e5b78 00007fa8ac0ef5c1 00007fa8ac0ef579 00007fa8ac0ef509
0321 02:59:33.005477 (+ 58760us) raft_consensus.cc:1685] finished
0321 02:59:33.005505 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:33.020147 (+ 14642us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9067,"raft.run_cpu_time_us":74,"raft.run_wall_time_us":74,"spinlock_wait_cycles":358043008,"thread_start_us":160037,"threads_started":2,"wal-append.queue_time_us":146850}
I20260321 02:59:33.023622 11201 leader_election.cc:400] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Vote denied by peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285) with higher term. Message: Invalid argument: T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7.
I20260321 02:59:33.024123 11201 leader_election.cc:403] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260321 02:59:33.025286 11275 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:33.027496 11380 raft_consensus.cc:2376] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6. Re-sending same reply.
I20260321 02:59:32.965703 13564 raft_consensus.cc:2804] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:32.991608 13718 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
W20260321 02:59:33.060460 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:33.028872 13564 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:33.050081 13920 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:33.060775 13938 raft_consensus.cc:2804] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:33.062098 13664 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
W20260321 02:59:33.079646 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:33.007582 13923 consensus_queue.cc:260] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.066768 13564 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:33.085038 13779 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:33.097191 11275 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:33.076274 13938 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:33.076571 13959 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:33.091114 13889 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:33.109201 11200 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:33.112623 11200 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:33.073729 13920 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.117635 11200 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:33.101554 13938 raft_consensus.cc:3060] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:33.125643 11200 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:33.103562 11973 raft_consensus.cc:1275] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260321 02:59:33.101805 13865 raft_consensus.cc:2749] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:33.142357 11275 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:33.110925 13888 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:33.123383 13923 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:33.119856 13939 raft_consensus.cc:2749] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:33.122390 13750 raft_consensus.cc:2804] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:33.131349 13719 raft_consensus.cc:2804] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260321 02:59:33.218842 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 84889.4us
I20260321 02:59:33.147477 13553 raft_consensus.cc:2804] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:33.129168 13846 consensus_queue.cc:1059] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.089s
I20260321 02:59:33.156495 13782 raft_consensus.cc:3055] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:33.211395 13750 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:33.105363 13959 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:33.144645 13594 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.228616 13719 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:33.229676 13968 log.cc:927] Time spent T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.084s user 0.002s sys 0.000s
I20260321 02:59:33.282258 11378 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
W20260321 02:59:33.270931 13956 log.cc:927] Time spent T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20260321 02:59:33.264963 13870 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:33.269126 13782 raft_consensus.cc:740] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:33.271334 13750 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:33.156298 13888 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:33.281849 13846 raft_consensus.cc:2749] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:33.283087 13951 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:33.283254 13553 raft_consensus.cc:697] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:33.283571 13718 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:33.291688 13719 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:33.291258 13723 consensus_queue.cc:1048] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
W20260321 02:59:33.697731 11311 consensus_meta.cc:287] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.735s user 0.023s sys 0.039s
I20260321 02:59:33.699141 11311 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
W20260321 02:59:33.386242 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:33.707023 13564 consensus_meta.cc:287] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.608s user 0.007s sys 0.004s
W20260321 02:59:33.605029 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 340519us
I20260321 02:59:33.382226 11231 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 7.
W20260321 02:59:33.533304 11942 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:33.709637 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:33.672469 13664 consensus_meta.cc:287] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.591s user 0.011s sys 0.001s
W20260321 02:59:33.702107 11274 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.532s user 0.015s sys 0.015s
W20260321 02:59:33.713620 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:33.574573 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 428382us
W20260321 02:59:33.392204 13915 log.cc:927] Time spent T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 1.322s user 0.005s sys 0.012s
W20260321 02:59:33.563081 13739 consensus_meta.cc:287] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.751s user 0.005s sys 0.009s
W20260321 02:59:33.351589 13967 log.cc:927] Time spent T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.231s user 0.000s sys 0.000s
W20260321 02:59:33.722299 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 580139us
W20260321 02:59:33.349471 13973 log.cc:927] Time spent T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260321 02:59:33.315677 13870 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:33.522039 13979 log.cc:927] Time spent T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.158s user 0.002s sys 0.000s
W20260321 02:59:33.475205 13977 log.cc:927] Time spent T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.176s user 0.000s sys 0.001s
I20260321 02:59:33.724736 13970 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:33.740862 13970 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.744215 13985 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:33.744987 13985 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.747442 13985 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:33.748343 13888 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:33.724241 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 497850us
W20260321 02:59:33.376724 13860 consensus_meta.cc:287] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.675s user 0.006s sys 0.006s
W20260321 02:59:33.731398 11309 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.008s sys 0.007s
W20260321 02:59:33.752818 11309 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.632s user 0.008s sys 0.007s
I20260321 02:59:33.631677 13594 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:33.378408 13959 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:33.411136 13920 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:33.754674 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:33.358448 13933 raft_consensus.cc:3055] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:33.753624 11275 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:33.470679 13961 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:33.754709 13959 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:33.757202 13959 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.701433 13779 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:33.725113 13664 raft_consensus.cc:2749] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285) with higher term. Message: Invalid argument: T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7.
I20260321 02:59:33.756191 13933 raft_consensus.cc:740] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:33.351112 13893 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:33.392525 13958 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:33.757077 13961 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.729598 13923 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:33.724838 13863 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
W20260321 02:59:33.722795 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:33.743602 13970 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:33.769191 11275 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:33.769492 13870 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:33.769304 13958 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.344568 13951 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.354329 13924 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.767583 13950 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:33.768455 13933 consensus_queue.cc:260] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.768771 13863 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.711472 13564 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:33.373389 13782 consensus_queue.cc:260] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.761636 13985 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:33.785861 11383 raft_consensus.cc:1240] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:33.771926 13958 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:33.734637 13739 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.504546 13553 consensus_queue.cc:237] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:33.772082 13920 raft_consensus.cc:2749] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:33.801301 13920 raft_consensus.cc:2749] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:33.796854 11308 raft_consensus.cc:1240] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
W20260321 02:59:33.781936 13938 consensus_meta.cc:287] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.636s user 0.010s sys 0.003s
I20260321 02:59:33.767494 13779 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:33.763005 13860 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.779829 13923 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.814581 13938 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.783209 13950 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.812872 13933 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:33.812229 13863 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:33.800575 13985 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.855811 13981 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:33.863341 13993 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:33.865280 13739 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:33.865520 13959 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:33.871603 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 159302us
W20260321 02:59:33.872018 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 63531.2us
I20260321 02:59:33.873556 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:33.877241 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 106835us
I20260321 02:59:33.877943 11275 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:33.865954 13983 consensus_queue.cc:1048] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
I20260321 02:59:33.882002 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:33.889868 11975 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:33.884110 13938 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:33.885282 13985 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:33.885306 13950 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:33.880118 13893 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:33.886471 13924 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:33.885355 13980 raft_consensus.cc:2804] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:33.892627 11275 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:33.902045 13980 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:33.904438 13980 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:33.905083 13779 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:33.905576 11200 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:33.906533 11200 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:33.878336 13564 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:33.907644 11200 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:33.906586 13923 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:33.908994 13860 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:33.864715 13981 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.911149 13981 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:33.911870 13951 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:32.615298 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:33.912117 13950 raft_consensus.cc:2764] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election decision vote started in defunct term 8: won
I20260321 02:59:33.908639 13863 raft_consensus.cc:2804] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:33.918179 13863 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:33.865574 13993 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:33.918583 13863 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:33.920280 13993 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:33.912793 13664 consensus_queue.cc:1048] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
W20260321 02:59:33.922705 11133 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44688 (request call id 130) took 13897 ms (client timeout 15000 ms). Trace:
W20260321 02:59:33.916842 13992 log.cc:927] Time spent T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.116s user 0.001s sys 0.000s
I20260321 02:59:33.926394 11275 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:33.927776 11275 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:33.930996 13863 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:33.931572 11349 leader_election.cc:336] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.672s (SENT)
I20260321 02:59:33.936265 11349 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:33.938213 13993 raft_consensus.cc:2749] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260321 02:59:33.901538 13719 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:33.940456 13961 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:33.935057 13782 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:33.938787 11308 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1713) took 1155 ms. Trace:
I20260321 02:59:33.941167 11308 rpcz_store.cc:276] 0321 02:59:32.783996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:32.784180 (+ 184us) service_pool.cc:225] Handling call
0321 02:59:32.838779 (+ 54599us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:33.803517 (+964738us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:33.938532 (+135015us) spinlock_profiling.cc:243] Waited 958 ms on lock 0x7b54001982d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:33.938727 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2202843520}
I20260321 02:59:33.942858 13924 raft_consensus.cc:2804] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:33.943265 13924 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:33.940727 13719 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:33.944129 13938 raft_consensus.cc:2749] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260321 02:59:33.923125 11133 rpcz_store.cc:269] 0321 02:59:20.025049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:20.025304 (+ 255us) service_pool.cc:225] Handling call
0321 02:59:33.922645 (+13897341us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:59:30.604980 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:59:30.614553 (+ 9573us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:59:30.614611 (+ 58us) write_op.cc:435] Acquired schema lock
0321 02:59:30.614628 (+ 17us) tablet.cc:661] Decoding operations
0321 02:59:30.615499 (+ 871us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:59:30.615604 (+ 105us) write_op.cc:641] Partition lock acquired for write op
0321 02:59:30.615635 (+ 31us) tablet.cc:684] Acquiring locks for 5 operations
0321 02:59:30.616684 (+ 1049us) tablet.cc:700] Row locks acquired
0321 02:59:30.616707 (+ 23us) write_op.cc:260] PREPARE: finished
0321 02:59:30.616906 (+ 199us) write_op.cc:270] Start()
0321 02:59:30.617058 (+ 152us) write_op.cc:276] Timestamp: P: 1774061970616833 usec, L: 0
0321 02:59:30.617078 (+ 20us) op_driver.cc:348] REPLICATION: starting
0321 02:59:30.621609 (+ 4531us) log.cc:844] Serialized 2035 byte log entry
0321 02:59:31.303737 (+682128us) op_driver.cc:464] REPLICATION: finished
0321 02:59:31.640399 (+336662us) write_op.cc:301] APPLY: starting
0321 02:59:31.649694 (+ 9295us) tablet.cc:1366] starting BulkCheckPresence
0321 02:59:31.649894 (+ 200us) tablet.cc:1369] finished BulkCheckPresence
0321 02:59:31.650068 (+ 174us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:59:31.693216 (+ 43148us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:59:31.811828 (+118612us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0321 02:59:31.820537 (+ 8709us) write_op.cc:312] APPLY: finished
0321 02:59:31.939655 (+119118us) log.cc:844] Serialized 47 byte log entry
0321 02:59:31.959399 (+ 19744us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:31.973582 (+ 14183us) write_op.cc:454] Released schema lock
0321 02:59:31.973675 (+ 93us) write_op.cc:341] FINISH: Updating metrics
0321 02:59:32.080227 (+106552us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:32.084466 (+ 4239us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2869275,"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":276589,"apply.run_cpu_time_us":4564,"apply.run_wall_time_us":443955,"num_ops":5,"prepare.queue_time_us":431412,"prepare.run_cpu_time_us":10500,"prepare.run_wall_time_us":172993,"raft.queue_time_us":250645,"raft.run_cpu_time_us":20113,"raft.run_wall_time_us":100004,"replication_time_us":686398,"thread_start_us":708952,"threads_started":4,"wal-append.queue_time_us":145438,"wal-append.run_cpu_time_us":12805,"wal-append.run_wall_time_us":2267491}]]}
I20260321 02:59:33.943683 13924 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:33.942015 11231 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59202 (request call id 225) took 1214 ms. Trace:
W20260321 02:59:33.960345 13987 log.cc:927] Time spent T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.107s user 0.002s sys 0.000s
I20260321 02:59:33.960667 11231 rpcz_store.cc:276] 0321 02:59:32.728503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:32.728667 (+ 164us) service_pool.cc:225] Handling call
0321 02:59:33.941824 (+1213157us) spinlock_profiling.cc:243] Waited 627 ms on lock 0x7b54002f99d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:33.941940 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1441916416}
I20260321 02:59:33.962168 13863 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:33.963003 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:33.963690 11383 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 6.
W20260321 02:59:33.966416 11942 leader_election.cc:336] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.762s (SENT)
I20260321 02:59:33.967012 11942 leader_election.cc:304] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:33.967928 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 60329us
I20260321 02:59:33.945946 11275 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:33.978221 11228 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59202 (request call id 226) took 1198 ms. Trace:
W20260321 02:59:33.978624 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 53873us
W20260321 02:59:33.941342 13750 consensus_meta.cc:287] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.606s user 0.005s sys 0.005s
I20260321 02:59:33.994138 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:33.978747 11228 rpcz_store.cc:276] 0321 02:59:32.768026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:32.768175 (+ 149us) service_pool.cc:225] Handling call
0321 02:59:32.813247 (+ 45072us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:32.849071 (+ 35824us) raft_consensus.cc:1534] Early marking committed up to index 7
0321 02:59:33.101413 (+252342us) log.cc:844] Serialized 14 byte log entry
0321 02:59:33.594086 (+492673us) log.cc:844] Serialized 14 byte log entry
0321 02:59:33.594454 (+ 368us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:33.631753 (+ 37299us) log.cc:844] Serialized 26 byte log entry
0321 02:59:33.632190 (+ 437us) raft_consensus.cc:1645] Marking committed up to 7
0321 02:59:33.632244 (+ 54us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:33.632566 (+ 322us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:33.904991 (+272425us) spinlock_profiling.cc:243] Waited 3.95 ms on lock 0x7b18004182a0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b2300441 00007fa8b230cca1 00007fa8a9484376 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b14721a1 00007fa8b1469237 00007fa8b14690ca 00007fa8b156092f
0321 02:59:33.908561 (+ 3570us) spinlock_profiling.cc:243] Waited 9.91 ms on lock 0x7b48000027d8. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b14721a1 00007fa8b1469237 00007fa8b14690ca 00007fa8b156092f 00007fa8b15605f7 00007fa8b1574c1c 00007fa8b1574bc8 00007fa8b1574b48 00007fa8b1574b03 00007fa8b1573dac 00007fa8b04acdf6
0321 02:59:33.959584 (+ 51023us) spinlock_profiling.cc:243] Waited 1.19 ms on lock 0x7b440002304c. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93c2dc5 00007fa8a93c37e2 00007fa8ac0f757b 00007fa8ac0f2e4f 00007fa8ac0e6c0e 00007fa8ac0e5f56 00007fa8ac0e5b78 00007fa8ac0ef5c1 00007fa8ac0ef579 00007fa8ac0ef509
0321 02:59:33.966009 (+ 6425us) raft_consensus.cc:1685] finished
0321 02:59:33.966044 (+ 35us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:33.966208 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16276,"prepare.queue_time_us":97202,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":92,"raft.queue_time_us":35928,"raft.run_cpu_time_us":13089,"raft.run_wall_time_us":348286,"spinlock_wait_cycles":34610048,"thread_start_us":672140,"threads_started":2,"wal-append.queue_time_us":703056}
W20260321 02:59:33.999909 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 54062.3us
I20260321 02:59:33.978274 13723 raft_consensus.cc:2804] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election won for term 9
I20260321 02:59:34.023826 11275 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:34.026490 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:33.977365 13593 raft_consensus.cc:2804] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:34.015511 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:34.016952 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:34.030839 11234 raft_consensus.cc:2410] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260321 02:59:34.023298 13971 log.cc:927] Time spent T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.811s user 0.012s sys 0.000s
I20260321 02:59:34.034211 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:34.039927 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:34.040704 11236 raft_consensus.cc:2368] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 8. Current term is 9.
I20260321 02:59:34.041193 11305 raft_consensus.cc:1240] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 4. Ops: []
W20260321 02:59:34.041735 13895 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.685s user 0.029s sys 0.020s
W20260321 02:59:34.042102 13895 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s user 0.029s sys 0.020s
I20260321 02:59:34.042428 11310 raft_consensus.cc:1240] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:34.044900 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:34.045857 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:34.047418 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:34.035985 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:34.024725 11275 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:34.036422 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:33.980683 13880 raft_consensus.cc:2749] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:34.036780 11309 raft_consensus.cc:1275] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260321 02:59:34.029807 11230 raft_consensus.cc:2468] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:34.049971 11229 raft_consensus.cc:2410] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 9 because replica has last-logged OpId of term: 8 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260321 02:59:34.050361 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:34.050515 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:34.050979 11384 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:34.051136 11378 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:34.053028 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:34.052798 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:34.054219 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:34.054518 11385 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:34.049117 11275 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:34.055442 11975 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:34.056850 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:34.056850 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:34.057561 11385 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:34.057709 11305 raft_consensus.cc:1240] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:34.058553 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:34.059108 11382 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:34.060662 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:34.062179 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:34.062686 11975 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:34.052793 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:34.064657 11308 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:34.066666 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:34.060892 11379 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:34.069938 11942 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:34.070375 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:34.072199 11379 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:34.071027 11275 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:34.073518 11275 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:34.038061 13938 raft_consensus.cc:2749] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:34.074829 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:34.077519 11275 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:34.077708 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:34.078526 11385 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:34.081773 11378 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:34.046576 13989 raft_consensus.cc:2804] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:34.008114 13750 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:34.082782 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:34.090934 11233 raft_consensus.cc:2468] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:34.090973 11350 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:34.087059 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:34.087127 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:34.088091 11349 leader_election.cc:336] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.715s (SENT)
I20260321 02:59:34.092226 11200 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:34.092478 11380 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 9
I20260321 02:59:34.092491 11349 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:34.039459 13951 consensus_queue.cc:1059] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.044s
I20260321 02:59:34.040314 13950 consensus_queue.cc:1048] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.049s
I20260321 02:59:34.058894 13846 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260321 02:59:34.554208 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 462003us
I20260321 02:59:34.573097 11200 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:34.160486 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:34.553938 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:34.554669 11308 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:34.282627 11384 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
I20260321 02:59:34.554100 11232 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 7.
I20260321 02:59:34.075632 13723 raft_consensus.cc:697] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:34.096694 11275 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:34.082852 13567 raft_consensus.cc:2804] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:34.277590 11231 raft_consensus.cc:1240] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 7. Current term is 8. Ops: []
W20260321 02:59:34.407696 14000 log.cc:927] Time spent T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.322s user 0.001s sys 0.000s
I20260321 02:59:34.087880 13877 raft_consensus.cc:2804] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260321 02:59:34.107368 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:34.171617 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:34.584177 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:34.190954 11380 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
W20260321 02:59:34.561475 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 473362us
I20260321 02:59:34.565596 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:34.560945 11381 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1707) took 1638 ms. Trace:
I20260321 02:59:34.574271 11377 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:34.183493 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:34.578198 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 481530us
I20260321 02:59:34.595737 11376 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:34.596155 11275 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:34.086637 13923 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:34.575313 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:34.140565 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:34.600404 11227 raft_consensus.cc:2468] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 5.
I20260321 02:59:34.600433 11308 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
W20260321 02:59:34.360031 13998 log.cc:927] Time spent T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.190s user 0.000s sys 0.001s
W20260321 02:59:34.581964 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:34.089131 13989 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:34.538877 11379 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
I20260321 02:59:34.306241 13970 raft_consensus.cc:2804] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:34.328269 13718 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:34.090766 13860 raft_consensus.cc:2804] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:34.598577 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:34.598690 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:34.589414 11381 rpcz_store.cc:276] 0321 02:59:32.922698 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:32.922858 (+ 160us) service_pool.cc:225] Handling call
0321 02:59:32.952811 (+ 29953us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:32.965997 (+ 13186us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:32.966044 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:32.966615 (+ 571us) log.cc:844] Serialized 26 byte log entry
0321 02:59:33.101470 (+134855us) raft_consensus.cc:1645] Marking committed up to 3
0321 02:59:33.102591 (+ 1121us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:33.102844 (+ 253us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:34.022821 (+919977us) spinlock_profiling.cc:243] Waited 8.28 ms on lock 0x7b2400266498. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93088f4 00007fa8a93069fb 00007fa8a9395799 00007fa8a9395425 00007fa8a9394a6d 00007fa8ac0cf4ae 00007fa8ac0e6c2c 00007fa8ac0e5f56 00007fa8ac0e5b78 00007fa8ac0ef5c1
0321 02:59:34.560586 (+537765us) raft_consensus.cc:1685] finished
0321 02:59:34.560619 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:34.560867 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8253,"raft.queue_time_us":106112,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":7799,"spinlock_wait_cycles":19049728,"thread_start_us":121739,"threads_started":1,"wal-append.queue_time_us":120725}
I20260321 02:59:34.579222 13865 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:34.604259 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:34.144630 13593 raft_consensus.cc:697] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:34.570513 13950 consensus_queue.cc:1243] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer fef7f8cc55b043039131bd9b2229f0a5 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260321 02:59:34.183912 13997 raft_consensus.cc:2749] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:34.069572 13870 consensus_queue.cc:1059] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.508s
I20260321 02:59:34.331099 13888 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:34.128675 13961 raft_consensus.cc:2804] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260321 02:59:34.609019 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 513305us
W20260321 02:59:34.580076 13980 consensus_meta.cc:287] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.675s user 0.009s sys 0.001s
I20260321 02:59:34.698385 13980 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:34.700075 13961 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:34.700897 13961 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:34.701431 13980 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:34.665622 11303 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
I20260321 02:59:34.703861 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:34.705003 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:34.577430 11233 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:34.706238 11233 raft_consensus.cc:606] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:34.717103 13984 raft_consensus.cc:3055] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:34.717593 13984 raft_consensus.cc:740] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:34.719276 11942 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:34.724824 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:34.728333 11232 raft_consensus.cc:1275] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260321 02:59:34.579929 13939 raft_consensus.cc:2804] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:34.609225 11200 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:34.603072 11276 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:34.734748 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 131876us
I20260321 02:59:34.578269 13929 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:34.736722 11230 raft_consensus.cc:1275] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260321 02:59:34.673436 11305 raft_consensus.cc:3060] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:34.671341 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:34.676381 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:34.738137 11308 raft_consensus.cc:3060] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 9
I20260321 02:59:34.580072 13564 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:34.581079 13567 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:34.608919 13970 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:34.611125 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:34.599588 13923 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:34.758667 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:34.598397 13993 raft_consensus.cc:2749] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:34.604626 13958 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:34.684547 13750 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:34.606958 13782 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:34.602133 13877 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:34.611703 13873 raft_consensus.cc:2804] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:34.684746 11134 catalog_manager.cc:5671] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 3 to 5, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 5 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:34.686933 13593 consensus_queue.cc:237] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:34.691602 11382 consensus_meta.cc:287] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.628s user 0.006s sys 0.000s
I20260321 02:59:34.696549 13723 consensus_queue.cc:237] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:34.819622 11235 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:34.666742 13860 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:34.582686 13938 raft_consensus.cc:2804] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:34.718353 13984 consensus_queue.cc:260] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:34.742874 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:34.743243 13564 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:34.744388 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:34.840912 11232 raft_consensus.cc:2368] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6.
I20260321 02:59:34.744803 13929 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:34.753978 11308 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
I20260321 02:59:34.734102 13888 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:34.614946 13989 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:34.757220 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:34.760496 11229 raft_consensus.cc:1240] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:34.671128 13718 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:34.791460 11236 raft_consensus.cc:1240] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:34.829377 11235 raft_consensus.cc:606] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:34.599152 13951 consensus_queue.cc:1048] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.176s
I20260321 02:59:34.740617 13980 raft_consensus.cc:2804] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:34.833341 13938 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:34.911526 13938 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:34.830109 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:34.917680 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:34.918859 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:34.919390 11304 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:34.748180 13988 raft_consensus.cc:3055] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:34.886567 13997 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:34.920786 13997 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:34.831676 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:34.832456 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:34.739575 13939 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:34.677951 13865 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:34.845388 11134 catalog_manager.cc:5671] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:34.828686 11382 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
W20260321 02:59:34.849045 11275 connection.cc:232] RPC call timeout handler was delayed by -1.3301530199998979s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1519s for a response
W20260321 02:59:34.922924 11275 connection.cc:232] RPC call timeout handler was delayed by -1.328353019999895s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1521s for a response
W20260321 02:59:34.923259 11275 connection.cc:232] RPC call timeout handler was delayed by -1.32295302s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1527s for a response
I20260321 02:59:34.759284 13970 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:34.789053 13870 raft_consensus.cc:2804] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:34.848335 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:34.831686 13933 consensus_meta.cc:287] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.967s user 0.011s sys 0.000s
I20260321 02:59:34.887995 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:34.759125 13923 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:34.874185 11305 raft_consensus.cc:2468] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
I20260321 02:59:34.753041 13567 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:34.759521 13895 raft_consensus.cc:2804] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:34.894301 11235 raft_consensus.cc:614] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:34.895378 11230 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:34.828995 13873 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:34.839030 13877 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:34.903578 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:34.771734 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 162604us
I20260321 02:59:34.942284 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:34.942826 11200 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:34.833345 13860 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:34.905655 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:34.850085 13950 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:34.754446 11376 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
I20260321 02:59:34.927093 13939 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
W20260321 02:59:34.908495 14003 log.cc:927] Time spent T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.190s user 0.000s sys 0.000s
W20260321 02:59:34.923554 11275 connection.cc:232] RPC call timeout handler was delayed by -1.2374530199999754s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1622s for a response
I20260321 02:59:34.931908 13984 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
W20260321 02:59:34.930320 11349 leader_election.cc:336] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.721s (SENT)
W20260321 02:59:34.840196 13924 consensus_meta.cc:287] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.873s user 0.009s sys 0.001s
I20260321 02:59:34.932390 11232 raft_consensus.cc:2393] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 8: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
W20260321 02:59:34.932816 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 178797us
I20260321 02:59:34.938617 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:34.943732 11234 consensus_queue.cc:237] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:59:34.954643 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:34.840327 13782 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:34.943064 11308 raft_consensus.cc:2393] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 6: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:34.956084 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:34.920311 13988 raft_consensus.cc:740] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:34.920331 13718 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:34.923532 11134 catalog_manager.cc:5671] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:34.945940 13873 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:34.976387 11134 catalog_manager.cc:5671] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 5 to 6, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 6 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:34.940055 13870 raft_consensus.cc:697] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:34.847260 13958 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:34.949075 11275 connection.cc:232] RPC call timeout handler was delayed by -1.1573530199999595s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17110000000000003s for a response
I20260321 02:59:34.950829 11200 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:34.985520 11275 connection.cc:232] RPC call timeout handler was delayed by -1.1465530199999421s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
I20260321 02:59:34.943845 11302 raft_consensus.cc:2393] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 5: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:34.932544 13980 raft_consensus.cc:697] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:34.952935 11349 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:34.969329 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:34.951535 13863 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:34.987577 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:34.949158 13895 raft_consensus.cc:697] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:34.951155 13733 raft_consensus.cc:2749] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:35.012881 13870 consensus_queue.cc:237] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:34.980499 11134 catalog_manager.cc:5671] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:35.014470 11134 catalog_manager.cc:5671] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 4 to 5, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 5 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:35.014569 13895 consensus_queue.cc:237] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:35.016755 11134 catalog_manager.cc:5671] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:35.018293 11134 catalog_manager.cc:5671] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 5 to 6, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 6 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:34.991402 13929 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:34.991771 11275 leader_election.cc:304] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:35.025094 11275 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:35.025213 13863 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:35.026592 13933 raft_consensus.cc:2749] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:35.027266 13863 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:35.027590 13933 raft_consensus.cc:2749] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:34.993916 11200 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:35.006009 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:34.956487 13924 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:34.983974 11974 raft_consensus.cc:1275] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260321 02:59:35.030311 13924 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:34.992588 13997 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:35.037534 11200 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:35.006157 13988 consensus_queue.cc:260] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:35.008353 13782 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:35.009197 13980 consensus_queue.cc:237] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:35.011047 13888 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:34.956203 11942 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:35.040084 11134 catalog_manager.cc:5671] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:35.042271 11231 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:35.060338 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:35.061398 11232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "513757b127054285b0686cd45044c596"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:35.062083 11232 raft_consensus.cc:606] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:34.951033 13950 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:35.061976 11134 catalog_manager.cc:5671] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 4 to 5, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 5 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:35.067600 11134 catalog_manager.cc:5671] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 1 to 3, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 3 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:59:34.991822 13750 raft_consensus.cc:2749] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:35.038434 11200 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:35.030283 13984 raft_consensus.cc:2804] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:35.061072 13951 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260321 02:59:35.108654 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:35.122558 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:35.124271 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 85896.6us
I20260321 02:59:35.065318 14008 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:35.130115 11306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1740) took 1046 ms. Trace:
I20260321 02:59:35.131248 11309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1733) took 1076 ms. Trace:
I20260321 02:59:35.073053 13993 raft_consensus.cc:2749] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:35.075814 14016 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:35.144172 11200 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:35.140911 11306 rpcz_store.cc:276] 0321 02:59:34.083904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:34.085117 (+ 1213us) service_pool.cc:225] Handling call
0321 02:59:34.715471 (+630354us) spinlock_profiling.cc:243] Waited 5.36 ms on lock 0x7b5800002a98. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b1522680 00007fa8b150cfe9 00007fa8b150ca92 00007fa8aaa8ad34
0321 02:59:34.715706 (+ 235us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:34.715912 (+ 206us) raft_consensus.cc:1534] Early marking committed up to index 4
0321 02:59:34.742316 (+ 26404us) log.cc:844] Serialized 14 byte log entry
0321 02:59:35.051204 (+308888us) log.cc:844] Serialized 14 byte log entry
0321 02:59:35.129393 (+ 78189us) spinlock_profiling.cc:243] Waited 5.88 ms on lock 0x7b1800360ca0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b2300441 00007fa8b230cca1 00007fa8a942a63a 00007fa8b2aa65c9 00007fa8a92e8c50 00007fa8a94b32b4 00007fa8a94b3225 00007fa8ac544440 00007fa8ac556af6 00007fa8ac567667 00007fa8ac5675e8
0321 02:59:35.129683 (+ 290us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:35.129740 (+ 57us) raft_consensus.cc:1645] Marking committed up to 4
0321 02:59:35.129786 (+ 46us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:35.129868 (+ 82us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:35.130016 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":68119,"prepare.queue_time_us":34092,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":84152,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":6796,"spinlock_wait_cycles":38632064,"thread_start_us":307037,"threads_started":1}
I20260321 02:59:35.101043 13846 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:35.156266 13984 raft_consensus.cc:697] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:35.124652 13988 raft_consensus.cc:3060] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:35.142817 11309 rpcz_store.cc:276] 0321 02:59:34.055326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:34.055458 (+ 132us) service_pool.cc:225] Handling call
0321 02:59:34.075235 (+ 19777us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:34.573153 (+497918us) raft_consensus.cc:1534] Early marking committed up to index 4
0321 02:59:34.573406 (+ 253us) log.cc:844] Serialized 14 byte log entry
0321 02:59:34.784225 (+210819us) spinlock_profiling.cc:243] Waited 3.35 ms on lock 0x7b18004442e0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b2300441 00007fa8b230cca1 00007fa8a9498d17 00007fa8b2276b19 00007fa8a94b546b 00007fa8a94b370f 00007fa8a94b59d2 00007fa8ac0e55ab 00007fa8ac0eb0c3 00007fa8ac0eb5da 00007fa8ac556e0a
0321 02:59:35.098158 (+313933us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:35.098231 (+ 73us) raft_consensus.cc:1645] Marking committed up to 4
0321 02:59:35.098292 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:35.131025 (+ 32733us) spinlock_profiling.cc:243] Waited 498 ms on lock 0x7b540018c754. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:35.131073 (+ 48us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:35.131193 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":291,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":5994,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":6017,"spinlock_wait_cycles":1152228864,"thread_start_us":313611,"threads_started":2,"wal-append.queue_time_us":498333}
I20260321 02:59:35.180037 11200 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:35.101922 13951 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
W20260321 02:59:35.190121 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 132524us
I20260321 02:59:35.196036 11307 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1741) took 1111 ms. Trace:
W20260321 02:59:35.196141 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 110371us
I20260321 02:59:35.135489 13564 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:35.209174 11200 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:35.209501 11309 raft_consensus.cc:1240] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [term 4 LEADER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 3. Current term is 4. Ops: []
I20260321 02:59:35.196664 11307 rpcz_store.cc:276] 0321 02:59:34.084585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:34.084706 (+ 121us) service_pool.cc:225] Handling call
0321 02:59:34.609099 (+524393us) spinlock_profiling.cc:243] Waited 5.65 ms on lock 0x7b5800002a98. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b1522680 00007fa8b150cfe9 00007fa8b150ca92 00007fa8aaa8ad34
0321 02:59:34.609382 (+ 283us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:34.609660 (+ 278us) raft_consensus.cc:1534] Early marking committed up to index 4
0321 02:59:34.677397 (+ 67737us) log.cc:844] Serialized 14 byte log entry
0321 02:59:35.034193 (+356796us) spinlock_profiling.cc:243] Waited 2.87 ms on lock 0x7b4800061a58. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b14721a1 00007fa8b1469237 00007fa8b14690ca 00007fa8b156092f 00007fa8b15605f7 00007fa8b1574c1c 00007fa8b1574bc8 00007fa8b1574b48 00007fa8b1574b03 00007fa8b1573dac 00007fa8b04acdf6
0321 02:59:35.195372 (+161179us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:35.195442 (+ 70us) raft_consensus.cc:1645] Marking committed up to 4
0321 02:59:35.195493 (+ 51us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:35.195663 (+ 170us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:35.195972 (+ 309us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8620,"prepare.queue_time_us":70451,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":10630,"raft.run_cpu_time_us":2298,"raft.run_wall_time_us":88077,"spinlock_wait_cycles":19599104,"thread_start_us":497012,"threads_started":2,"wal-append.queue_time_us":450502}
I20260321 02:59:35.140116 13886 raft_consensus.cc:2804] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:35.214495 13886 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:35.214946 13886 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:35.187544 13723 raft_consensus.cc:2804] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:35.190984 13846 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:35.223770 11200 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:35.228739 11200 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:35.229908 11232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:35.198318 13951 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:35.230753 11232 raft_consensus.cc:606] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:35.238441 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:35.239399 11975 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:35.244064 11276 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:35.224721 13723 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:35.202175 13865 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:35.248174 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:35.258456 11200 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:35.185565 14008 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:35.220378 13929 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260321 02:59:35.237802 13750 raft_consensus.cc:2804] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:35.274471 13733 raft_consensus.cc:2804] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election won for term 9
I20260321 02:59:35.247874 13723 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:35.188750 14016 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:35.293256 11308 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1744) took 1177 ms. Trace:
I20260321 02:59:35.274634 13718 raft_consensus.cc:2804] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election won for term 9
W20260321 02:59:35.294584 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 365196us
W20260321 02:59:35.291909 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 88008.3us
I20260321 02:59:35.230862 13983 consensus_queue.cc:1048] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
I20260321 02:59:35.277758 14019 raft_consensus.cc:2804] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260321 02:59:35.300765 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 274557us
I20260321 02:59:35.288993 13929 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:35.294479 13553 raft_consensus.cc:993] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb: : Instructing follower fef7f8cc55b043039131bd9b2229f0a5 to start an election
I20260321 02:59:35.227676 13938 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:35.343652 11232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:35.344684 11275 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:35.315478 13799 raft_consensus.cc:2804] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:35.293515 13970 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:35.356424 11304 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 7.
I20260321 02:59:35.362977 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:35.344858 14019 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:35.361146 11275 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:35.341846 13750 raft_consensus.cc:697] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:35.313266 13733 raft_consensus.cc:697] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:35.352000 13929 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:35.352061 13950 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:35.376648 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 118315us
I20260321 02:59:35.379603 11275 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:35.360335 13958 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:35.385833 11200 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:35.362798 13799 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:35.365542 13924 raft_consensus.cc:2749] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260321 02:59:35.394933 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:35.368168 13664 raft_consensus.cc:1081] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Signalling peer fef7f8cc55b043039131bd9b2229f0a5 to start an election
W20260321 02:59:35.381376 14015 log.cc:927] Time spent T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.249s user 0.002s sys 0.000s
I20260321 02:59:35.357213 13877 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:35.487284 11200 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:35.488834 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 181721us
I20260321 02:59:35.431490 11227 raft_consensus.cc:2393] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 7: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:35.492107 13924 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:35.370031 13567 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:35.386437 14020 raft_consensus.cc:2764] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
W20260321 02:59:35.508563 14013 log.cc:927] Time spent T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.234s user 0.000s sys 0.001s
W20260321 02:59:35.510052 14030 log.cc:927] Time spent T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.144s user 0.001s sys 0.000s
I20260321 02:59:35.424595 13983 raft_consensus.cc:2804] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election won for term 9
I20260321 02:59:35.422703 13750 consensus_queue.cc:237] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:35.317261 13984 consensus_queue.cc:237] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:35.518051 11200 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:35.392423 13997 raft_consensus.cc:2764] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
W20260321 02:59:35.516443 14018 log.cc:927] Time spent T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.175s user 0.000s sys 0.001s
I20260321 02:59:35.392012 13718 raft_consensus.cc:697] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:35.375758 14019 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
W20260321 02:59:35.556296 13961 consensus_meta.cc:287] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.855s user 0.009s sys 0.005s
I20260321 02:59:35.488456 13799 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
W20260321 02:59:35.527576 11349 leader_election.cc:336] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.784s (SENT)
I20260321 02:59:35.576424 11349 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:35.577478 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 49947.2us
W20260321 02:59:35.578210 11349 leader_election.cc:336] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.840s (SENT)
I20260321 02:59:35.578779 11349 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:35.422719 13865 consensus_queue.cc:1048] T a42977f759be4144a5862197a77e395d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.112s
I20260321 02:59:35.551192 11305 raft_consensus.cc:1240] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:35.525157 13886 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:35.536034 13924 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:35.532892 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 144443us
W20260321 02:59:35.579820 11349 leader_election.cc:336] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.877s (SENT)
I20260321 02:59:35.600420 11349 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:35.601440 11349 leader_election.cc:336] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.881s (SENT)
I20260321 02:59:35.554100 13983 raft_consensus.cc:697] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:35.596501 13888 raft_consensus.cc:2804] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election won for term 6
W20260321 02:59:35.582255 13939 consensus_meta.cc:287] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.627s user 0.005s sys 0.005s
I20260321 02:59:35.596833 13718 consensus_queue.cc:237] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:35.546794 13733 consensus_queue.cc:237] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:35.602052 11349 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:35.547818 13989 consensus_meta.cc:287] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.654s user 0.011s sys 0.000s
I20260321 02:59:35.604030 11275 leader_election.cc:400] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6.
I20260321 02:59:35.619354 11275 leader_election.cc:403] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260321 02:59:35.597158 14016 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:35.602573 13923 consensus_meta.cc:287] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.658s user 0.013s sys 0.000s
I20260321 02:59:35.599911 13958 raft_consensus.cc:2749] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260321 02:59:35.604527 14032 log.cc:927] Time spent T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.092s user 0.002s sys 0.000s
W20260321 02:59:35.623682 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:35.595120 14021 raft_consensus.cc:2749] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:35.610530 13993 raft_consensus.cc:2749] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:35.631805 13950 consensus_queue.cc:1059] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260321 02:59:35.631479 13923 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:35.583038 13961 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:35.630077 13860 consensus_meta.cc:287] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.661s user 0.008s sys 0.004s
I20260321 02:59:35.631620 14016 raft_consensus.cc:2749] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:35.643457 11311 raft_consensus.cc:1240] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:35.678303 11306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1748) took 1382 ms. Trace:
I20260321 02:59:35.677896 11304 raft_consensus.cc:1240] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:35.679024 11306 rpcz_store.cc:276] 0321 02:59:34.298610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:35.179595 (+880985us) service_pool.cc:225] Handling call
0321 02:59:35.180065 (+ 470us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:35.192737 (+ 12672us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:35.192784 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:35.193448 (+ 664us) log.cc:844] Serialized 26 byte log entry
0321 02:59:35.354575 (+161127us) raft_consensus.cc:1645] Marking committed up to 3
0321 02:59:35.362167 (+ 7592us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:35.370863 (+ 8696us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:35.678054 (+307191us) raft_consensus.cc:1685] finished
0321 02:59:35.678077 (+ 23us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:35.678223 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20989,"raft.queue_time_us":138,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":13345,"thread_start_us":121900,"threads_started":1,"wal-append.queue_time_us":356570}
I20260321 02:59:35.682430 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:35.653118 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:35.672175 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:35.629261 14020 raft_consensus.cc:2749] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6.
I20260321 02:59:35.688607 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:35.629045 13938 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:35.616658 13939 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:35.685014 11384 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:35.632193 13970 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:35.627482 13989 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:35.618672 14024 consensus_queue.cc:1048] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.108s
I20260321 02:59:35.689422 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:35.720481 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:35.758029 11306 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:35.759783 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:35.731185 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:35.761062 11304 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:35.636504 13950 raft_consensus.cc:3055] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Stepping down as leader of term 8
I20260321 02:59:35.757218 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:35.760165 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:35.735606 11377 raft_consensus.cc:1275] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260321 02:59:35.759384 11306 raft_consensus.cc:1240] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:35.768757 11306 raft_consensus.cc:1240] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:35.640333 13567 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:35.784962 13950 raft_consensus.cc:740] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:35.806135 14021 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:35.645035 13664 raft_consensus.cc:2804] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election won for term 9
I20260321 02:59:35.840378 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:35.674870 13997 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:35.639483 14008 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:35.848652 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 222584us
I20260321 02:59:35.849002 11310 raft_consensus.cc:2393] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 9: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:35.843091 11227 raft_consensus.cc:2468] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
W20260321 02:59:35.736624 14029 log.cc:927] Time spent T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
W20260321 02:59:35.850302 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 248872us
I20260321 02:59:35.716895 13553 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:35.681079 13877 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:35.857245 11349 leader_election.cc:336] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.941s (SENT)
I20260321 02:59:35.860337 11233 raft_consensus.cc:1240] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:35.735978 13923 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:35.867864 11349 leader_election.cc:336] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.945s (SENT)
I20260321 02:59:35.869536 11382 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "513757b127054285b0686cd45044c596"
dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:40934
I20260321 02:59:35.869905 11349 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:35.870414 11382 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:59:35.870976 11382 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
W20260321 02:59:35.923812 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 56000.5us
W20260321 02:59:35.860391 14025 log.cc:927] Time spent T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.300s user 0.000s sys 0.001s
I20260321 02:59:35.641535 13980 consensus_queue.cc:1059] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.215s
I20260321 02:59:35.849697 11275 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:35.944097 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 184392us
W20260321 02:59:35.954144 11349 leader_election.cc:336] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.966s (SENT)
I20260321 02:59:35.687553 13860 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:35.855578 13888 raft_consensus.cc:697] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:35.969149 11304 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:35.961279 11349 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:35.971652 11349 leader_election.cc:336] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.990s (SENT)
I20260321 02:59:35.972168 11349 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:35.966953 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 117312us
I20260321 02:59:35.970475 11304 raft_consensus.cc:1240] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: []
I20260321 02:59:35.985038 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:35.976565 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:36.003278 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:36.004356 11384 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:35.983923 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:36.006314 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:36.005633 11379 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:35.689443 13983 consensus_queue.cc:237] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:36.006040 11275 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:35.858122 13997 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:36.014261 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 706044us
I20260321 02:59:36.021005 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:36.023872 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 62443.6us
W20260321 02:59:35.962236 14032 log.cc:927] Time spent T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20260321 02:59:36.018535 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:35.973649 11382 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:35.850157 14021 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:35.968837 13567 raft_consensus.cc:476] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Not starting pre-election -- already a leader
I20260321 02:59:35.956656 14060 raft_consensus.cc:2749] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:36.086820 13939 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:36.087272 11311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1737) took 2017 ms. Trace:
I20260321 02:59:36.089701 11311 rpcz_store.cc:276] 0321 02:59:34.069436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:34.071716 (+ 2280us) service_pool.cc:225] Handling call
0321 02:59:34.666374 (+594658us) spinlock_profiling.cc:243] Waited 5.5 ms on lock 0x7b5800002a98. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b1522680 00007fa8b150cfe9 00007fa8b150ca92 00007fa8aaa8ad34
0321 02:59:34.666566 (+ 192us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:35.678419 (+1011853us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:36.086971 (+408552us) spinlock_profiling.cc:243] Waited 977 ms on lock 0x7b54003390d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:36.087188 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2258763904}
I20260321 02:59:35.996588 14016 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
W20260321 02:59:35.975287 13846 consensus_meta.cc:287] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.724s user 0.007s sys 0.004s
I20260321 02:59:36.089742 11382 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:35.996407 13664 raft_consensus.cc:697] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:35.971735 13895 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:36.102018 14008 consensus_queue.cc:1048] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:59:36.102941 14067 consensus_queue.cc:1059] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:59:36.003751 13970 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:36.005463 13923 raft_consensus.cc:2804] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:35.872164 13553 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:36.122501 11305 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1746) took 1944 ms. Trace:
I20260321 02:59:36.004633 13993 raft_consensus.cc:2749] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:36.124727 11233 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58214 (request call id 1340) took 1402 ms. Trace:
W20260321 02:59:36.120275 11940 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:36.006516 13924 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:36.105713 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:36.059749 13886 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:36.144639 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 138654us
W20260321 02:59:36.277537 11309 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.749s user 0.005s sys 0.004s
W20260321 02:59:36.277964 11309 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.749s user 0.005s sys 0.004s
W20260321 02:59:36.256690 11349 leader_election.cc:336] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.535s (SENT)
I20260321 02:59:36.256182 11200 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:36.279223 11349 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:35.303925 11308 rpcz_store.cc:276] 0321 02:59:34.115899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:34.973409 (+857510us) service_pool.cc:225] Handling call
0321 02:59:34.973733 (+ 324us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:34.973919 (+ 186us) raft_consensus.cc:1534] Early marking committed up to index 4
0321 02:59:34.984167 (+ 10248us) log.cc:844] Serialized 14 byte log entry
0321 02:59:35.038265 (+ 54098us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x7b18000453a0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b2300441 00007fa8b230cca1 00007fa8a9498d17 00007fa8b2276b19 00007fa8a94b546b 00007fa8a94b370f 00007fa8a94b59d2 00007fa8ac0e55ab 00007fa8ac0eb0c3 00007fa8ac0eb5da 00007fa8ac556e0a
0321 02:59:35.292638 (+254373us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:35.292752 (+ 114us) raft_consensus.cc:1645] Marking committed up to 4
0321 02:59:35.292832 (+ 80us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:35.292957 (+ 125us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:35.293169 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2430,"prepare.queue_time_us":10075,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":4119,"raft.queue_time_us":2580,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":5441,"spinlock_wait_cycles":4401024,"thread_start_us":251109,"threads_started":1}
I20260321 02:59:36.099201 14020 raft_consensus.cc:3055] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:36.317057 11309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49786 (request call id 235) took 1635 ms. Trace:
I20260321 02:59:36.310750 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:36.275990 11305 rpcz_store.cc:276] 0321 02:59:34.178268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:34.978842 (+800574us) service_pool.cc:225] Handling call
0321 02:59:34.979204 (+ 362us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:35.582309 (+603105us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:36.122290 (+539981us) spinlock_profiling.cc:243] Waited 572 ms on lock 0x7b540018c254. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:36.122446 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1314883456}
I20260321 02:59:36.276166 11233 rpcz_store.cc:276] 0321 02:59:34.722398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:34.722602 (+ 204us) service_pool.cc:225] Handling call
0321 02:59:34.742066 (+ 19464us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:35.867489 (+1125423us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:36.124400 (+256911us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400333bd4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:36.124650 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2564711296}
I20260321 02:59:36.327914 11309 rpcz_store.cc:276] 0321 02:59:34.681801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:35.384509 (+702708us) service_pool.cc:225] Handling call
0321 02:59:35.465561 (+ 81052us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:35.465884 (+ 323us) raft_consensus.cc:1534] Early marking committed up to index 7
0321 02:59:35.524782 (+ 58898us) log.cc:844] Serialized 14 byte log entry
0321 02:59:36.316640 (+791858us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:36.316703 (+ 63us) raft_consensus.cc:1645] Marking committed up to 7
0321 02:59:36.316773 (+ 70us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:36.316840 (+ 67us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:36.316985 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38128,"prepare.queue_time_us":210232,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":28762,"raft.queue_time_us":38826,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":108,"thread_start_us":749752,"threads_started":1}
I20260321 02:59:36.304833 11384 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:36.329620 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:36.284855 11304 raft_consensus.cc:1240] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:36.283720 11978 raft_consensus.cc:1275] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260321 02:59:36.325462 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:36.249282 14021 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:36.330855 11305 raft_consensus.cc:2393] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 8: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
W20260321 02:59:36.238982 14026 log.cc:927] Time spent T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.599s user 0.001s sys 0.000s
I20260321 02:59:36.332909 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:36.356544 11379 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:36.357491 11381 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:35.967587 13950 consensus_queue.cc:260] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:36.362126 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:36.363610 11384 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:36.363880 11234 raft_consensus.cc:2393] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
I20260321 02:59:36.365229 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:36.370293 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:36.244882 13988 consensus_meta.cc:287] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 1.046s user 0.011s sys 0.000s
I20260321 02:59:36.159075 13567 raft_consensus.cc:2804] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:36.377866 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:36.150774 13961 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:36.397431 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:36.399024 11304 raft_consensus.cc:2410] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260321 02:59:36.275597 13923 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:36.253235 14057 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:36.445149 14038 log.cc:927] Time spent T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.708s user 0.001s sys 0.000s
I20260321 02:59:36.373106 14021 raft_consensus.cc:2749] T 2b701d6135bc458c9aef1d5303344525 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:36.446357 14021 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:36.446539 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:36.446849 14021 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:36.334393 14020 raft_consensus.cc:740] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:36.447889 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:36.449067 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:36.436825 14050 log.cc:927] Time spent T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.468s user 0.000s sys 0.001s
I20260321 02:59:36.450109 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:36.451150 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:36.470796 11305 raft_consensus.cc:2393] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:36.472064 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:36.473058 13860 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:36.473205 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:36.474263 11305 raft_consensus.cc:2393] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 8: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:36.475178 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:36.476516 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:36.477444 11305 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:36.477074 13664 consensus_queue.cc:237] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:36.478293 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:36.479635 11381 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:36.484715 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:36.442994 14027 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:36.316885 13970 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:36.444294 14066 log.cc:927] Time spent T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.295s user 0.001s sys 0.000s
I20260321 02:59:36.333841 14067 raft_consensus.cc:3055] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:36.149611 14040 consensus_queue.cc:1059] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.258s
I20260321 02:59:36.149449 13846 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:36.448593 14020 consensus_queue.cc:260] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:36.452224 14021 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:36.397311 13989 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:36.445111 14057 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:36.436961 13567 raft_consensus.cc:697] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:36.463651 13997 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:36.468212 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:36.522766 11383 raft_consensus.cc:3060] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:36.408542 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:36.524183 11311 raft_consensus.cc:2368] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 8. Current term is 9.
I20260321 02:59:36.439291 13553 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:36.532944 11302 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1747) took 2354 ms. Trace:
I20260321 02:59:36.536098 14049 raft_consensus.cc:2749] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260321 02:59:36.537779 14046 log.cc:927] Time spent T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.162s user 0.001s sys 0.000s
W20260321 02:59:36.441969 14065 log.cc:927] Time spent T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.156s user 0.002s sys 0.000s
I20260321 02:59:36.544682 14020 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:36.505962 14027 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:36.549121 13567 consensus_queue.cc:237] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:36.488133 11380 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.119s sys 0.156s
I20260321 02:59:36.485949 11381 raft_consensus.cc:2393] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 7: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
W20260321 02:59:36.550526 11380 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s user 0.120s sys 0.156s
I20260321 02:59:36.551431 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:36.552628 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:36.553210 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:36.553540 11380 raft_consensus.cc:1275] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260321 02:59:36.554718 11380 raft_consensus.cc:1240] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:36.555500 14057 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:36.556066 11381 raft_consensus.cc:1240] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9. Ops: []
W20260321 02:59:36.558856 13873 consensus_meta.cc:287] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 1.575s user 0.012s sys 0.000s
I20260321 02:59:36.559338 13873 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:36.561589 11303 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 6.
W20260321 02:59:36.562798 11303 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49786 (request call id 237) took 1865 ms (client timeout 1709 ms). Trace:
W20260321 02:59:36.563146 11303 rpcz_store.cc:269] 0321 02:59:34.697728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:34.704771 (+ 7043us) service_pool.cc:225] Handling call
0321 02:59:36.562579 (+1857808us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b540018cc54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:36.562695 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4268371200}
I20260321 02:59:36.564392 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:36.565198 11303 raft_consensus.cc:2393] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 4: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
W20260321 02:59:36.565201 13929 consensus_meta.cc:287] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 1.173s user 0.010s sys 0.001s
I20260321 02:59:36.448499 11307 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49786 (request call id 229) took 1846 ms. Trace:
I20260321 02:59:36.492848 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:36.567363 11379 raft_consensus.cc:2376] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6. Re-sending same reply.
I20260321 02:59:36.493427 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:36.494423 11384 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:36.348608 13993 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
W20260321 02:59:36.495255 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 239158us
I20260321 02:59:36.445231 13923 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:36.159183 13895 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:36.507988 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:36.850728 11309 raft_consensus.cc:2393] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
W20260321 02:59:36.507802 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 251121us
W20260321 02:59:36.851370 11349 leader_election.cc:336] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.579s (SENT)
W20260321 02:59:36.509418 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:36.453410 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:36.491894 13723 consensus_meta.cc:287] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 1.150s user 0.009s sys 0.003s
W20260321 02:59:36.489768 13799 consensus_meta.cc:287] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.893s user 0.008s sys 0.003s
I20260321 02:59:36.413061 13880 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:36.379482 13924 consensus_queue.cc:1048] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.131s
I20260321 02:59:36.506827 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:36.860682 11304 raft_consensus.cc:2393] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:36.529117 14070 consensus_queue.cc:1048] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.039s
I20260321 02:59:36.494587 14067 raft_consensus.cc:740] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:36.543721 14016 raft_consensus.cc:2749] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:36.549902 14027 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:36.881466 11377 raft_consensus.cc:1275] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260321 02:59:36.317979 14008 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:36.884438 14008 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:36.759641 11383 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
W20260321 02:59:36.665498 11376 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.928s user 0.000s sys 0.012s
I20260321 02:59:36.843766 11308 raft_consensus.cc:2368] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 7. Current term is 9.
W20260321 02:59:36.889051 11376 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.152s user 0.008s sys 0.019s
I20260321 02:59:36.567178 11307 rpcz_store.cc:276] 0321 02:59:34.602182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:35.223450 (+621268us) service_pool.cc:225] Handling call
0321 02:59:35.240674 (+ 17224us) raft_consensus.cc:1471] Updating replica for 2 ops
0321 02:59:35.241060 (+ 386us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:35.241103 (+ 43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0321 02:59:35.241818 (+ 715us) log.cc:844] Serialized 52 byte log entry
0321 02:59:35.368292 (+126474us) raft_consensus.cc:1645] Marking committed up to 8
0321 02:59:35.382795 (+ 14503us) log.cc:844] Serialized 14 byte log entry
0321 02:59:35.417525 (+ 34730us) log.cc:844] Serialized 14 byte log entry
0321 02:59:35.417851 (+ 326us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:35.417971 (+ 120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:36.436619 (+1018648us) spinlock_profiling.cc:243] Waited 5.71 ms on lock 0x7b18003ab8e0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b2300441 00007fa8b230cca1 00007fa8a94b61c3 00007fa8a94b8591 00007fa8a94b8549 00007fa8a94b84d9 00007fa8a94b84a1 00007fa8a94b779d 0000000000379254 0000000000379099 00007fa8a949974a
0321 02:59:36.448195 (+ 11576us) raft_consensus.cc:1685] finished
0321 02:59:36.448229 (+ 34us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:36.448423 (+ 194us) inbound_call.cc:173] Queueing success response
0321 02:59:36.558141 (+109718us) spinlock_profiling.cc:243] Waited 35.7 ms on lock 0x7b18003ab8e0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b2300441 00007fa8b230cca1 00007fa8a94b61c3 00007fa8a94b8591 00007fa8a94b8549 00007fa8a94b84d9 00007fa8a94b84a1 00007fa8a94b779d 0000000000379254 0000000000379099 00007fa8a949974a
Metrics: {"mutex_wait_us":26903,"prepare.queue_time_us":1194012,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":72,"raft.queue_time_us":18761,"raft.run_cpu_time_us":518,"raft.run_wall_time_us":62177,"spinlock_wait_cycles":95252608,"thread_start_us":73513,"threads_started":1,"wal-append.queue_time_us":360834}
W20260321 02:59:36.580569 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 257248us
I20260321 02:59:36.889689 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:36.890560 11942 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 2.133s (SENT) [suppressed 49 similar messages]
I20260321 02:59:36.630825 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:36.533568 11302 rpcz_store.cc:276] 0321 02:59:34.178615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:35.036987 (+858372us) service_pool.cc:225] Handling call
0321 02:59:35.037346 (+ 359us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:35.130122 (+ 92776us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:35.130152 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:35.130677 (+ 525us) log.cc:844] Serialized 26 byte log entry
0321 02:59:35.311558 (+180881us) raft_consensus.cc:1645] Marking committed up to 7
0321 02:59:35.538882 (+227324us) log.cc:844] Serialized 14 byte log entry
0321 02:59:35.540954 (+ 2072us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:36.532372 (+991418us) spinlock_profiling.cc:243] Waited 51.8 ms on lock 0x7b540020ba54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:36.532444 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:36.532507 (+ 63us) raft_consensus.cc:1685] finished
0321 02:59:36.532645 (+ 138us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:36.532858 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":44254,"prepare.queue_time_us":45585,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":44123,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":7388,"spinlock_wait_cycles":119034624,"thread_start_us":364186,"threads_started":2,"wal-append.queue_time_us":302844}
I20260321 02:59:36.893220 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:36.894482 11302 raft_consensus.cc:2410] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260321 02:59:36.277570 13888 consensus_queue.cc:237] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:36.845597 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:36.851754 11349 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:36.855593 13933 raft_consensus.cc:493] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:36.856132 14024 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:36.645658 13951 consensus_meta.cc:287] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 1.371s user 0.014s sys 0.000s
I20260321 02:59:36.566084 13929 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:36.954325 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 65459.4us
I20260321 02:59:36.733618 13970 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:36.687191 13715 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:36.864898 13880 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:36.862609 13723 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:36.872430 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:36.877375 13950 raft_consensus.cc:3060] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:36.880209 13846 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:36.881820 14073 log.cc:927] Time spent T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
W20260321 02:59:36.618402 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:36.636341 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:36.886962 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:37.042953 11309 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:36.858532 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:36.902410 11942 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:37.044131 11303 raft_consensus.cc:2393] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 8: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
W20260321 02:59:36.891683 14069 log.cc:927] Time spent T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.153s user 0.001s sys 0.000s
W20260321 02:59:36.954396 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 60908.4us
W20260321 02:59:36.806495 11940 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:36.854601 13993 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:37.020609 11382 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:37.119642 11382 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40950 (request call id 1295) took 1119 ms. Trace:
I20260321 02:59:37.120060 11382 rpcz_store.cc:276] 0321 02:59:36.000150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:36.839427 (+839277us) service_pool.cc:225] Handling call
0321 02:59:36.872254 (+ 32827us) spinlock_profiling.cc:243] Waited 2.4 ms on lock 0x7b58002e3298. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b150e760 00007fa8b150e044 00007fa8aaa8f834 00007fa8aaa8f7c2
0321 02:59:37.119573 (+247319us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5512576}
I20260321 02:59:37.023669 11232 raft_consensus.cc:606] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:36.914639 14031 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.004s sys 0.007s
I20260321 02:59:37.015421 13715 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:37.044915 11942 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
W20260321 02:59:37.133306 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 88487.2us
I20260321 02:59:36.976006 13865 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:37.121417 11232 raft_consensus.cc:614] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:36.945868 13933 raft_consensus.cc:515] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:36.976658 13863 consensus_queue.cc:1059] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.144s
I20260321 02:59:36.945556 14024 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:36.862468 13799 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:37.156139 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:37.160550 11200 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:36.888839 13873 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:37.161180 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 309828us
I20260321 02:59:37.164172 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:37.075824 14021 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:37.130510 14031 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.010s user 0.004s sys 0.011s
W20260321 02:59:37.186602 11349 leader_election.cc:336] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.605s (SENT)
I20260321 02:59:37.188755 11978 raft_consensus.cc:2368] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7.
W20260321 02:59:37.192166 11349 leader_election.cc:336] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.673s (SENT)
I20260321 02:59:37.192792 11349 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:36.973714 13951 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:37.180217 14072 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:37.142881 13750 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:37.201316 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:37.201570 14074 raft_consensus.cc:2804] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:37.202899 14027 consensus_queue.cc:1059] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
W20260321 02:59:37.203349 14047 log.cc:927] Time spent T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 1.237s user 0.003s sys 0.001s
I20260321 02:59:37.203578 13970 consensus_queue.cc:1059] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:59:37.204159 11276 leader_election.cc:400] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Vote denied by peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285) with higher term. Message: Invalid argument: T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7.
I20260321 02:59:37.235625 11276 leader_election.cc:403] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260321 02:59:37.205587 11200 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:37.208140 11201 consensus_peers.cc:597] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f4ec8d11b7b4e6096d8c2b8ece078dc. This is attempt 1: this message will repeat every 5th retry.
W20260321 02:59:37.236543 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 47761.4us
I20260321 02:59:37.208801 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:37.238790 11385 raft_consensus.cc:2368] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6.
I20260321 02:59:37.239372 11385 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40950 (request call id 1292) took 1241 ms. Trace:
W20260321 02:59:37.209172 11310 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56630 (request call id 1734) took 3152 ms (client timeout 1901 ms). Trace:
I20260321 02:59:37.240558 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:37.240464 11310 rpcz_store.cc:269] 0321 02:59:34.056349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:34.056654 (+ 305us) service_pool.cc:225] Handling call
0321 02:59:37.208862 (+3152208us) spinlock_profiling.cc:243] Waited 1.79 s on lock 0x7b54000e79d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:37.209046 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4119388416}
I20260321 02:59:37.241520 11311 raft_consensus.cc:2468] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 3.
I20260321 02:59:37.241999 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:37.243221 11310 raft_consensus.cc:2393] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
W20260321 02:59:37.212442 14080 log.cc:927] Time spent T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.469s user 0.001s sys 0.000s
I20260321 02:59:37.214684 14024 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:37.155519 14043 raft_consensus.cc:2804] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:37.207752 11232 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:41224 (request call id 43) took 1904 ms. Trace:
I20260321 02:59:37.225706 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:37.217020 11974 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45078 (request call id 508) took 1128 ms. Trace:
W20260321 02:59:37.191358 14019 consensus_meta.cc:287] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 1.597s user 0.008s sys 0.012s
I20260321 02:59:37.216295 13880 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:37.232640 14074 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:37.249101 13873 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:37.264173 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:37.249276 13895 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:37.198055 13733 raft_consensus.cc:3055] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:37.239816 11385 rpcz_store.cc:276] 0321 02:59:35.997774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:37.208429 (+1210655us) service_pool.cc:225] Handling call
0321 02:59:37.239297 (+ 30868us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:37.245735 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:37.197346 13567 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:37.156668 13865 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:37.015192 14067 consensus_queue.cc:260] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:37.195712 14076 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:37.225584 13593 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:37.258358 11232 rpcz_store.cc:276] 0321 02:59:35.303020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:35.303187 (+ 167us) service_pool.cc:225] Handling call
0321 02:59:37.207438 (+1904251us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b54002f8ad4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac546c33 00007fa8b1510cca 00007fa8aaa9d934 00007fa8aaa9d8c2 00007fa8aaa9d7f1 00007fa8aaa9d76c 00007fa8aaa9ca22 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:37.207626 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3835471104}
I20260321 02:59:37.234652 14072 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:37.267215 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:37.272740 13970 raft_consensus.cc:3055] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:37.237350 13718 raft_consensus.cc:3055] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:37.259538 11974 rpcz_store.cc:276] 0321 02:59:36.088177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:36.100674 (+ 12497us) service_pool.cc:225] Handling call
0321 02:59:36.101031 (+ 357us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:36.252371 (+151340us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:36.252404 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:36.253005 (+ 601us) log.cc:844] Serialized 26 byte log entry
0321 02:59:36.508701 (+255696us) raft_consensus.cc:1645] Marking committed up to 5
0321 02:59:36.508762 (+ 61us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:36.508854 (+ 92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:37.216727 (+707873us) raft_consensus.cc:1685] finished
0321 02:59:37.216757 (+ 30us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:37.216940 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":150913,"raft.queue_time_us":151029,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"thread_start_us":255221,"threads_started":1,"wal-append.queue_time_us":386391}
I20260321 02:59:37.277562 11310 raft_consensus.cc:2393] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 9: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:37.198565 14021 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:37.279153 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 73639.3us
I20260321 02:59:37.286798 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:37.258275 14043 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:37.273295 13799 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:37.297236 13863 consensus_queue.cc:1059] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:59:37.297358 14067 raft_consensus.cc:3060] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:37.277222 13733 raft_consensus.cc:740] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:37.297457 13715 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:37.298282 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260321 02:59:37.264240 14074 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:37.235723 13750 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:37.300729 11308 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:37.284606 13984 raft_consensus.cc:2804] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:37.373481 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:37.306813 11311 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:37.375931 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:37.307056 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:59:37.380977 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:37.381681 11384 raft_consensus.cc:1275] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260321 02:59:37.342824 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:37.281244 13723 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:37.288190 13970 raft_consensus.cc:740] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:37.290383 13718 raft_consensus.cc:740] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:37.393788 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:37.305250 14044 consensus_queue.cc:1059] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.094s
I20260321 02:59:37.394888 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:37.296041 14043 raft_consensus.cc:3060] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:37.396163 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:37.397313 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:37.397524 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:37.290902 14008 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:37.274384 14019 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:37.394894 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:37.429334 11303 raft_consensus.cc:2393] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:37.295346 13929 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 10 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:37.290231 13593 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:37.290004 14076 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:37.417315 11231 raft_consensus.cc:2468] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 4.
I20260321 02:59:37.417845 11200 leader_election.cc:304] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:37.420080 11377 raft_consensus.cc:1240] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:37.420156 11384 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:37.420802 11385 raft_consensus.cc:2393] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 7: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:37.420624 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:37.284247 13567 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:37.425383 13865 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:37.391275 14040 raft_consensus.cc:3055] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:37.449620 11306 raft_consensus.cc:1240] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:37.402282 11383 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:37.442377 11200 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:37.444072 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:37.446352 11384 raft_consensus.cc:1275] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260321 02:59:37.446856 11385 raft_consensus.cc:1275] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260321 02:59:37.452450 11200 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:37.452677 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:37.447400 11381 raft_consensus.cc:2393] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 7: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:37.421499 14036 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:37.455184 11308 raft_consensus.cc:2393] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:37.452842 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:37.487816 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:37.488773 11303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56616 (request call id 1764) took 1012 ms. Trace:
I20260321 02:59:37.489005 11381 raft_consensus.cc:2393] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 7: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:37.489254 11303 rpcz_store.cc:276] 0321 02:59:36.475898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:37.437607 (+961709us) service_pool.cc:225] Handling call
0321 02:59:37.488687 (+ 51080us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:37.463529 11377 raft_consensus.cc:1275] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260321 02:59:37.470678 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:37.471021 11310 raft_consensus.cc:2376] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6. Re-sending same reply.
I20260321 02:59:37.431519 14008 raft_consensus.cc:2749] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:37.456840 11308 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56616 (request call id 1746) took 1021 ms. Trace:
I20260321 02:59:37.496169 11384 raft_consensus.cc:1275] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260321 02:59:37.437263 13993 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:37.498847 11308 rpcz_store.cc:276] 0321 02:59:36.434175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:37.424108 (+989933us) service_pool.cc:225] Handling call
0321 02:59:37.455875 (+ 31767us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:37.440902 13886 raft_consensus.cc:3055] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:37.453394 14040 raft_consensus.cc:740] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:37.441967 13933 leader_election.cc:290] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:37.447060 14027 raft_consensus.cc:2749] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:37.514393 11380 raft_consensus.cc:1240] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:37.515614 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:37.518607 11377 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:37.470247 13664 raft_consensus.cc:2749] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:37.517416 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:37.520826 11381 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 10
I20260321 02:59:37.532809 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:37.533504 11385 raft_consensus.cc:1275] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260321 02:59:37.491696 13951 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:37.503896 13886 raft_consensus.cc:740] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:37.543673 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:37.549183 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:37.550115 11384 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:37.551353 11308 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56616 (request call id 1747) took 1116 ms. Trace:
I20260321 02:59:37.551883 11308 rpcz_store.cc:276] 0321 02:59:36.434553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:37.512834 (+1078281us) service_pool.cc:225] Handling call
0321 02:59:37.551268 (+ 38434us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:37.562976 11381 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 10.
I20260321 02:59:37.469525 13929 raft_consensus.cc:2749] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:37.564319 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:37.542636 14020 consensus_meta.cc:287] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.997s user 0.014s sys 0.000s
I20260321 02:59:37.577247 11383 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:37.587198 13886 consensus_queue.cc:260] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:37.487493 13718 consensus_queue.cc:260] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:37.497359 14036 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:37.689221 11376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1737) took 2011 ms. Trace:
I20260321 02:59:37.532943 14058 consensus_queue.cc:1059] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.302s
I20260321 02:59:37.616025 14072 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:37.618423 13567 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:37.593705 14021 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:37.637475 11305 raft_consensus.cc:2368] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 7. Current term is 8.
I20260321 02:59:37.651549 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:37.660702 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:37.454270 13970 consensus_queue.cc:260] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:37.692382 11350 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 388404us
I20260321 02:59:37.671121 14020 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:37.671350 13750 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:37.599059 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:37.589497 14040 consensus_queue.cc:260] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:37.699570 11376 rpcz_store.cc:276] 0321 02:59:35.677628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:35.683522 (+ 5894us) service_pool.cc:225] Handling call
0321 02:59:35.717453 (+ 33931us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:35.717703 (+ 250us) raft_consensus.cc:1534] Early marking committed up to index 4
0321 02:59:36.889584 (+1171881us) log.cc:844] Serialized 14 byte log entry
0321 02:59:37.686423 (+796839us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:37.686486 (+ 63us) raft_consensus.cc:1645] Marking committed up to 4
0321 02:59:37.688691 (+ 2205us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:37.688949 (+ 258us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:37.689140 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19241,"prepare.queue_time_us":618205,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":66,"raft.queue_time_us":226548,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":16644,"thread_start_us":1948619,"threads_started":3,"wal-append.queue_time_us":730464}
I20260321 02:59:37.684633 13593 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:37.699244 14085 log.cc:927] Time spent T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.269s user 0.000s sys 0.002s
I20260321 02:59:37.691264 13886 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:37.691562 14016 raft_consensus.cc:2749] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:37.709789 11975 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
W20260321 02:59:37.714083 11350 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:37.716753 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:37.717511 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:37.698835 14019 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:37.720543 11228 raft_consensus.cc:1240] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:37.719151 14076 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:37.721541 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:37.723595 11384 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
W20260321 02:59:37.723730 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 253125us
I20260321 02:59:37.723970 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:37.739538 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:37.742739 11308 raft_consensus.cc:2393] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 7: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:37.742683 11228 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
W20260321 02:59:37.710194 14084 log.cc:927] Time spent T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.256s user 0.001s sys 0.001s
I20260321 02:59:37.724993 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:37.725134 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:37.725642 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:37.725921 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:37.691265 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 499100us
I20260321 02:59:37.733001 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:37.720935 11303 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:37.744979 11303 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56616 (request call id 1766) took 1185 ms. Trace:
W20260321 02:59:37.745136 11349 leader_election.cc:336] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.748s (SENT)
I20260321 02:59:37.745337 11303 rpcz_store.cc:276] 0321 02:59:36.559749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:37.490134 (+930385us) service_pool.cc:225] Handling call
0321 02:59:37.744900 (+254766us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:37.745662 11349 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:37.746392 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:37.746598 11235 raft_consensus.cc:1240] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9. Ops: [8.5-8.5]
W20260321 02:59:37.746930 13923 consensus_meta.cc:287] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.883s user 0.008s sys 0.004s
I20260321 02:59:37.747418 13923 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:37.747818 14024 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:37.748425 14024 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:37.749833 13923 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:37.750965 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:37.740850 14043 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:37.751605 13567 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:37.752270 13567 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:37.753002 14043 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:37.753628 11974 raft_consensus.cc:1240] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:37.754634 13920 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:37.754922 14076 raft_consensus.cc:2749] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260321 02:59:37.755173 13920 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
W20260321 02:59:37.756265 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:37.728910 13895 consensus_queue.cc:1048] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
W20260321 02:59:37.761477 11275 connection.cc:232] RPC call timeout handler was delayed by -1.4165144590000409s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1543s for a response
I20260321 02:59:37.743944 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:37.757212 11306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1756) took 2109 ms. Trace:
W20260321 02:59:37.758513 11349 leader_election.cc:336] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.801s (SENT)
I20260321 02:59:37.763687 11349 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:37.763114 11306 rpcz_store.cc:276] 0321 02:59:35.648003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:35.772446 (+124443us) service_pool.cc:225] Handling call
0321 02:59:35.772797 (+ 351us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:37.450015 (+1677218us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:37.756975 (+306960us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b54001ac2d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:37.757150 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3855851392}
I20260321 02:59:37.765456 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:37.766225 11306 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
W20260321 02:59:37.766376 11349 leader_election.cc:336] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.995s (SENT)
I20260321 02:59:37.766978 11349 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:37.768786 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:37.770278 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:37.772249 11306 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:37.778538 11200 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:37.778594 11232 raft_consensus.cc:1240] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 7. Current term is 8. Ops: []
W20260321 02:59:37.778575 11275 connection.cc:232] RPC call timeout handler was delayed by -1.3742144589999725s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15900000000000003s for a response
I20260321 02:59:37.724169 13984 raft_consensus.cc:697] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:37.754418 13567 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:37.780539 11275 connection.cc:232] RPC call timeout handler was delayed by -1.3706144589999667s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15940000000000001s for a response
I20260321 02:59:37.782964 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:37.777367 13923 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:37.788570 13984 consensus_queue.cc:237] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:37.790699 14091 raft_consensus.cc:2749] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260321 02:59:37.750608 14024 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:37.792131 11306 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 8.
W20260321 02:59:37.792308 11275 connection.cc:232] RPC call timeout handler was delayed by -1.3706144589999667s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15940000000000001s for a response
W20260321 02:59:37.805725 11275 connection.cc:232] RPC call timeout handler was delayed by -1.3634144590000687s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1602s for a response
W20260321 02:59:37.806094 11275 connection.cc:232] RPC call timeout handler was delayed by -1.3319144590000178s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1637s for a response
I20260321 02:59:37.806129 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:37.806547 11275 connection.cc:232] RPC call timeout handler was delayed by -1.328314459000012s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16410000000000002s for a response
W20260321 02:59:37.806869 11275 connection.cc:232] RPC call timeout handler was delayed by -1.2509144590000005s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17270000000000002s for a response
I20260321 02:59:37.807093 11306 raft_consensus.cc:2393] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 9: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
W20260321 02:59:37.807149 11275 connection.cc:232] RPC call timeout handler was delayed by -1.1591144589999658s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1829s for a response
W20260321 02:59:37.807576 11275 connection.cc:232] RPC call timeout handler was delayed by -1.1339144590000387s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1857s for a response
I20260321 02:59:37.722918 14020 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:37.792173 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:37.778717 14076 raft_consensus.cc:2749] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260321 02:59:37.801038 11349 leader_election.cc:336] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 2.101s (SENT)
I20260321 02:59:37.792809 11308 raft_consensus.cc:2368] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 8. Current term is 9.
I20260321 02:59:37.808601 11349 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:37.808980 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:37.807870 11275 connection.cc:232] RPC call timeout handler was delayed by -1.1303144590000329s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18610000000000002s for a response
W20260321 02:59:37.810014 11275 connection.cc:232] RPC call timeout handler was delayed by -1.0997144589999834s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1895s for a response
W20260321 02:59:37.810370 11275 connection.cc:232] RPC call timeout handler was delayed by -1.0781144590000622s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19190000000000002s for a response
W20260321 02:59:37.810634 11275 connection.cc:232] RPC call timeout handler was delayed by -1.0484144590000142s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1952s for a response
I20260321 02:59:37.805322 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:37.811204 11275 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:37.812758 11275 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:37.821457 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:37.822233 11232 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 8
I20260321 02:59:37.822834 11978 raft_consensus.cc:2393] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
I20260321 02:59:37.821406 11275 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:37.796994 14036 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:37.808269 13567 consensus_queue.cc:1059] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.018s
I20260321 02:59:37.825901 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:37.778085 13593 consensus_queue.cc:1048] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260321 02:59:37.827770 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:37.828495 11275 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:37.830129 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:37.831182 11306 raft_consensus.cc:2368] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8.
I20260321 02:59:37.832315 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:37.833777 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:37.827039 14019 raft_consensus.cc:2749] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:37.835461 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:37.836390 11974 raft_consensus.cc:2468] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:37.839970 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:37.842677 11309 raft_consensus.cc:2393] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:37.846427 11309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56616 (request call id 1748) took 1411 ms. Trace:
I20260321 02:59:37.846766 11309 rpcz_store.cc:276] 0321 02:59:36.434882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:37.839233 (+1404351us) service_pool.cc:225] Handling call
0321 02:59:37.846351 (+ 7118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:37.842911 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:37.844448 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:37.846580 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:37.843070 11349 leader_election.cc:336] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 2.115s (SENT)
I20260321 02:59:37.850493 11349 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:37.845634 11377 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:37.855901 11303 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:37.820339 14021 raft_consensus.cc:2749] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260321 02:59:37.857905 13933 raft_consensus.cc:2749] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260321 02:59:37.857985 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:37.819484 13923 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:37.862003 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:37.862716 11381 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 6.
I20260321 02:59:37.863186 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:37.864965 11942 connection.cc:232] RPC call timeout handler was delayed by -1.7869657940000252s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22040000000000004s for a response
I20260321 02:59:37.865464 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:37.866050 11381 raft_consensus.cc:2468] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 5.
I20260321 02:59:37.860620 11306 raft_consensus.cc:2393] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 9: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:37.866782 11306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56616 (request call id 1759) took 1392 ms. Trace:
I20260321 02:59:37.867249 11306 rpcz_store.cc:276] 0321 02:59:36.474412 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:37.842558 (+1368146us) service_pool.cc:225] Handling call
0321 02:59:37.866711 (+ 24153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:37.868949 11306 raft_consensus.cc:1275] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260321 02:59:37.872287 11232 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:37.828402 13846 raft_consensus.cc:2749] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:37.827163 14098 raft_consensus.cc:2804] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:37.896477 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:37.896662 11377 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
W20260321 02:59:37.905933 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 50366.5us
W20260321 02:59:37.906973 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 63932.7us
W20260321 02:59:37.907775 11349 leader_election.cc:336] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 2.242s (SENT)
I20260321 02:59:37.908278 11349 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:37.898154 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:37.900270 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:37.928967 11309 raft_consensus.cc:2393] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 8: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:37.929107 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:37.870836 11306 raft_consensus.cc:1275] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260321 02:59:37.930176 11303 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 10.
I20260321 02:59:37.933182 11377 raft_consensus.cc:1275] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260321 02:59:37.934283 11308 raft_consensus.cc:1275] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260321 02:59:37.930598 13567 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:37.841050 14074 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:37.861994 14024 raft_consensus.cc:2749] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:37.853886 14036 consensus_queue.cc:1048] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
I20260321 02:59:37.904726 14098 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:37.841164 14095 consensus_queue.cc:1048] T 96159c887ede40b68c4dc0ab508ada79 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.093s
W20260321 02:59:37.911386 11349 leader_election.cc:336] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 2.275s (SENT)
I20260321 02:59:37.947042 11379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45634 (request call id 396) took 1623 ms. Trace:
I20260321 02:59:37.949791 11309 raft_consensus.cc:1275] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260321 02:59:37.988402 11349 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:37.991210 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 79852.8us
W20260321 02:59:37.992082 11349 leader_election.cc:336] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 2.412s (SENT)
I20260321 02:59:37.992542 11349 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:37.994238 11305 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56616 (request call id 1754) took 1521 ms (client timeout 1748 ms). Trace:
W20260321 02:59:37.994674 11305 rpcz_store.cc:269] 0321 02:59:36.472958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:36.478055 (+ 5097us) service_pool.cc:225] Handling call
0321 02:59:37.994008 (+1515953us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54000e74d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:37.994172 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2663677824}
I20260321 02:59:37.988448 11379 rpcz_store.cc:276] 0321 02:59:36.323207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:36.568222 (+245015us) service_pool.cc:225] Handling call
0321 02:59:36.568478 (+ 256us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:37.169765 (+601287us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:37.169812 (+ 47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:37.169874 (+ 62us) raft_consensus.cc:1645] Marking committed up to 8
0321 02:59:37.169921 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:37.946701 (+776780us) spinlock_profiling.cc:243] Waited 601 ms on lock 0x7b54000b8bd4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:37.946768 (+ 67us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:37.946962 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1382194048}
W20260321 02:59:37.945353 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 103820us
I20260321 02:59:37.902069 14089 raft_consensus.cc:2749] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260321 02:59:37.941354 14067 consensus_meta.cc:287] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.574s user 0.008s sys 0.004s
W20260321 02:59:37.996400 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 130923us
I20260321 02:59:37.999068 14106 consensus_queue.cc:1048] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:59:37.976344 13567 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:38.002321 13567 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:38.002821 14074 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:38.003479 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:37.994036 14091 raft_consensus.cc:2749] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:37.932842 14044 consensus_queue.cc:1048] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.057s
I20260321 02:59:38.010165 11229 raft_consensus.cc:2393] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 7: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
W20260321 02:59:38.016551 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 85083.7us
I20260321 02:59:38.017976 11349 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:38.020345 11349 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:37.989408 14098 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
W20260321 02:59:38.032689 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 204255us
W20260321 02:59:38.015766 11942 leader_election.cc:336] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 2.025s (SENT)
I20260321 02:59:38.035131 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.036110 11302 raft_consensus.cc:2368] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 5. Current term is 6.
I20260321 02:59:38.019551 14091 raft_consensus.cc:2749] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260321 02:59:38.021409 14089 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:38.023896 14072 raft_consensus.cc:2804] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:38.036495 11201 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:38.040555 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:38.051367 11303 raft_consensus.cc:2368] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 5. Current term is 6.
I20260321 02:59:38.023684 13993 raft_consensus.cc:2749] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:38.042026 11942 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:38.042330 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:38.042129 11200 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:38.043326 14020 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:38.043634 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.044765 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:38.046108 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.047180 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:38.048398 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.048851 14027 consensus_queue.cc:1048] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:59:38.049067 14024 consensus_queue.cc:1059] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260321 02:59:38.049247 11310 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56616 (request call id 1758) took 1575 ms (client timeout 2099 ms). Trace:
W20260321 02:59:38.066028 11310 rpcz_store.cc:269] 0321 02:59:36.474132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:37.286541 (+812409us) service_pool.cc:225] Handling call
0321 02:59:38.049072 (+762531us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b540018bd54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:38.049184 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":415203072}
I20260321 02:59:38.050179 11275 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
W20260321 02:59:38.055138 11349 leader_election.cc:336] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 2.002s (SENT)
W20260321 02:59:38.057065 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:38.067744 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:38.023605 14021 consensus_queue.cc:1059] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260321 02:59:38.064396 11200 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:38.068374 11349 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:38.062598 11309 raft_consensus.cc:2376] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 7. Re-sending same reply.
I20260321 02:59:38.072755 11275 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:38.048866 13723 raft_consensus.cc:3055] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Stepping down as leader of term 8
I20260321 02:59:38.057495 14072 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:38.061111 14043 raft_consensus.cc:2749] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260321 02:59:38.123096 11349 leader_election.cc:336] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 2.082s (SENT)
I20260321 02:59:38.065063 14096 raft_consensus.cc:2804] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260321 02:59:38.127341 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 63031.6us
I20260321 02:59:38.129020 11349 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:38.070308 14099 raft_consensus.cc:2804] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:38.069972 13993 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:38.129709 11200 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:38.130460 11349 leader_election.cc:336] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 2.099s (SENT)
I20260321 02:59:38.138917 11349 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:38.077555 13723 raft_consensus.cc:740] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:38.077793 13933 raft_consensus.cc:2749] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:38.080876 14021 raft_consensus.cc:2749] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260321 02:59:38.187693 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 114960us
I20260321 02:59:38.125387 14020 raft_consensus.cc:2804] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:38.057178 14089 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:38.130134 14091 raft_consensus.cc:3055] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 LEADER]: Stepping down as leader of term 7
W20260321 02:59:38.200657 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 71028.6us
I20260321 02:59:38.130364 14072 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:38.131234 14096 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:38.080377 13567 consensus_queue.cc:1059] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 8.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.117s
W20260321 02:59:38.210626 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 80187.5us
W20260321 02:59:38.212237 11349 leader_election.cc:336] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 2.242s (SENT)
I20260321 02:59:38.212936 11349 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:38.209815 14099 raft_consensus.cc:697] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:38.215051 13933 consensus_queue.cc:1059] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:59:38.192063 13923 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:38.217057 14021 raft_consensus.cc:2749] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260321 02:59:38.212747 13664 raft_consensus.cc:3055] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Stepping down as leader of term 4
I20260321 02:59:38.223623 13664 raft_consensus.cc:740] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:38.210006 14096 raft_consensus.cc:3060] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:38.224859 11200 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:38.223749 14021 raft_consensus.cc:2749] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:38.198973 14020 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:38.234046 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.197304 14107 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Leader election won for term 10
I20260321 02:59:38.208636 13593 raft_consensus.cc:2804] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election won for term 9
I20260321 02:59:38.236514 13895 consensus_queue.cc:1059] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260321 02:59:38.245977 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.247089 11232 raft_consensus.cc:2393] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 8: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
W20260321 02:59:38.247876 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:38.189812 13993 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:38.216806 14101 log.cc:927] Time spent T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.241s user 0.001s sys 0.000s
I20260321 02:59:38.206101 14091 raft_consensus.cc:740] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:38.237816 14095 raft_consensus.cc:2804] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:38.243240 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.190181 13733 consensus_queue.cc:260] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:38.251186 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.261261 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.261445 11980 raft_consensus.cc:1240] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:38.240738 14020 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:38.248787 13664 consensus_queue.cc:260] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:59:38.249387 14021 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:38.325623 14021 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:38.242309 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:38.328421 11229 raft_consensus.cc:3055] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:38.328852 11229 raft_consensus.cc:740] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:38.329070 13970 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:38.329815 11229 consensus_queue.cc:260] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:38.286692 14107 raft_consensus.cc:697] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:38.331480 14107 consensus_queue.cc:237] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:38.332594 13929 consensus_peers.cc:313] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Unable to generate Tablet Copy request for peer: Not found: Peer not tracked or queue not in leader mode.
I20260321 02:59:38.288352 13993 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:38.288244 14089 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:38.289253 13873 consensus_meta.cc:287] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 1.011s user 0.011s sys 0.002s
I20260321 02:59:38.336028 14019 raft_consensus.cc:3055] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:38.291237 11234 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 7.
I20260321 02:59:38.336599 14019 raft_consensus.cc:740] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:38.298445 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.258653 13567 raft_consensus.cc:3055] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Stepping down as leader of term 8
I20260321 02:59:38.337304 13567 raft_consensus.cc:740] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:38.337795 14104 raft_consensus.cc:3055] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Stepping down as leader of term 8
I20260321 02:59:38.338320 14104 raft_consensus.cc:740] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:38.338260 13567 consensus_queue.cc:260] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:38.340180 13846 raft_consensus.cc:3055] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:38.340648 13846 raft_consensus.cc:740] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:38.341513 13846 consensus_queue.cc:260] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:38.342902 13846 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:38.300420 14095 raft_consensus.cc:697] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
W20260321 02:59:38.302673 14110 log.cc:927] Time spent T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
I20260321 02:59:38.345731 13664 raft_consensus.cc:3060] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:38.346199 14095 consensus_queue.cc:237] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:38.305599 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:38.313896 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:38.313360 14091 consensus_queue.cc:260] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:38.350417 14091 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:38.314957 14099 consensus_queue.cc:237] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:38.355211 13920 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:38.318787 14024 consensus_queue.cc:1059] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260321 02:59:38.339488 14104 consensus_queue.cc:260] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.3, Last appended by leader: 3, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:38.320395 13950 consensus_meta.cc:287] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 1.286s user 0.014s sys 0.000s
I20260321 02:59:38.323526 13718 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:38.237723 13723 consensus_queue.cc:260] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:38.336109 13873 raft_consensus.cc:2749] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285) with higher term. Message: Invalid argument: T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7.
I20260321 02:59:38.336796 14021 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:38.378840 11304 raft_consensus.cc:2368] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 8. Current term is 9.
W20260321 02:59:38.380230 11304 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56616 (request call id 1741) took 1947 ms (client timeout 1990 ms). Trace:
I20260321 02:59:38.279749 13593 raft_consensus.cc:697] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:38.380889 11311 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 7.
I20260321 02:59:38.382021 13593 consensus_queue.cc:237] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:38.382457 11311 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56616 (request call id 1762) took 1907 ms (client timeout 2206 ms). Trace:
W20260321 02:59:38.383002 11311 rpcz_store.cc:269] 0321 02:59:36.475305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:37.375585 (+900280us) service_pool.cc:225] Handling call
0321 02:59:38.382238 (+1006653us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b540018cc54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:38.382381 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2309178240}
I20260321 02:59:38.357684 13733 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:38.360110 14040 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
W20260321 02:59:38.380677 11304 rpcz_store.cc:269] 0321 02:59:36.432444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:36.861210 (+428766us) service_pool.cc:225] Handling call
0321 02:59:38.380006 (+1518796us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54001d56d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:38.380163 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3422403328}
I20260321 02:59:38.236881 14117 raft_consensus.cc:2749] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:38.364233 14104 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:38.372263 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:38.372833 14102 log.cc:927] Time spent T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.384s user 0.004s sys 0.001s
I20260321 02:59:38.373106 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.375696 14019 consensus_queue.cc:260] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:38.396613 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 327417us
I20260321 02:59:38.390388 11378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1744) took 1018 ms. Trace:
I20260321 02:59:38.392680 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.388242 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:38.398540 11349 connection.cc:232] RPC call timeout handler was delayed by -1.2439636229998996s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32370000000000004s for a response
I20260321 02:59:38.401503 11349 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:38.386386 14112 log.cc:927] Time spent T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.159s user 0.004s sys 0.000s
I20260321 02:59:38.412492 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:38.415223 11377 pending_rounds.cc:85] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5: Aborting all ops after (but not including) 1
I20260321 02:59:38.415822 11377 pending_rounds.cc:107] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260321 02:59:38.416179 11377 raft_consensus.cc:2889] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:38.418478 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:38.419183 11379 raft_consensus.cc:2468] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
W20260321 02:59:38.389196 14114 log.cc:927] Time spent T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.097s user 0.000s sys 0.004s
I20260321 02:59:38.393172 14076 raft_consensus.cc:2749] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:38.422670 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:38.386925 13865 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:38.424789 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:38.431376 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:38.427280 11379 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:38.428332 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:38.433739 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:38.437297 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:38.437965 11376 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:38.421427 14089 raft_consensus.cc:2804] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:38.424100 14027 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:38.437024 14092 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.010s sys 0.003s
I20260321 02:59:38.432216 13723 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:38.446739 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.449641 11232 raft_consensus.cc:2393] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 9: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:38.447154 11977 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45098 (request call id 407) took 1282 ms. Trace:
W20260321 02:59:38.450651 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 253843us
I20260321 02:59:38.451181 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:38.453127 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.454651 11384 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:38.455060 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:38.455415 11379 raft_consensus.cc:1275] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260321 02:59:38.457297 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:38.457818 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:38.459682 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:38.459810 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:38.460436 11382 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:38.460790 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:38.461213 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:38.461320 11383 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:38.461804 11382 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:38.462592 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:38.463213 11382 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:38.463987 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:38.464932 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:38.465087 11349 leader_election.cc:336] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.760s (SENT)
I20260321 02:59:38.465447 11382 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:38.466094 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:38.467794 11381 raft_consensus.cc:2393] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 9: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:38.467308 11302 raft_consensus.cc:1275] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260321 02:59:38.473663 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:38.474138 11381 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:38.468652 11310 raft_consensus.cc:1240] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260321 02:59:38.477509 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:38.478505 11306 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:38.478587 11275 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:38.480993 11310 raft_consensus.cc:1240] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:38.479868 11275 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:38.477501 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:38.466495 11382 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:38.441890 14089 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:38.483496 11385 raft_consensus.cc:2368] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7.
I20260321 02:59:38.498482 11349 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:38.498936 11383 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:38.499684 14021 raft_consensus.cc:2749] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260321 02:59:38.501350 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 80577.1us
I20260321 02:59:38.502368 11349 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:38.502514 11275 leader_election.cc:304] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:38.434492 13865 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:38.483683 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:38.484011 11376 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
W20260321 02:59:38.446848 14092 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.515s user 0.010s sys 0.004s
I20260321 02:59:38.488722 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:38.496881 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.450481 11977 rpcz_store.cc:276] 0321 02:59:37.164904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:37.165047 (+ 143us) service_pool.cc:225] Handling call
0321 02:59:37.195421 (+ 30374us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:37.698165 (+502744us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:37.698213 (+ 48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:37.698851 (+ 638us) log.cc:844] Serialized 26 byte log entry
0321 02:59:37.784464 (+ 85613us) raft_consensus.cc:1645] Marking committed up to 6
0321 02:59:37.784591 (+ 127us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:37.784802 (+ 211us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:38.343086 (+558284us) spinlock_profiling.cc:243] Waited 9.47 ms on lock 0x7b24003151e8. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93088f4 00007fa8a93069fb 00007fa8a9395799 00007fa8a9395425 00007fa8a9394a6d 00007fa8ac0cf4ae 00007fa8ac0e6c2c 00007fa8ac0e5f56 00007fa8ac0e5b78 00007fa8ac0ef5c1
0321 02:59:38.443556 (+100470us) raft_consensus.cc:1685] finished
0321 02:59:38.443584 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:38.447087 (+ 3503us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":70170,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":78,"spinlock_wait_cycles":21789952,"thread_start_us":559353,"threads_started":2,"wal-append.queue_time_us":162625}
I20260321 02:59:38.399255 11378 rpcz_store.cc:276] 0321 02:59:37.371515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:37.372129 (+ 614us) service_pool.cc:225] Handling call
0321 02:59:37.372551 (+ 422us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:37.372894 (+ 343us) raft_consensus.cc:1534] Early marking committed up to index 4
0321 02:59:37.401667 (+ 28773us) log.cc:844] Serialized 14 byte log entry
0321 02:59:38.018061 (+616394us) spinlock_profiling.cc:243] Waited 9.21 ms on lock 0x7b180035c0e0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b2300441 00007fa8b230cca1 00007fa8a9498d17 00007fa8b2276b19 00007fa8a94b546b 00007fa8a94b370f 00007fa8a94b59d2 00007fa8ac0e55ab 00007fa8ac0eb0c3 00007fa8ac0eb5da 00007fa8ac556e0a
0321 02:59:38.389872 (+371811us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:38.389950 (+ 78us) raft_consensus.cc:1645] Marking committed up to 4
0321 02:59:38.390015 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:38.390134 (+ 119us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:38.390305 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11096,"prepare.queue_time_us":28514,"prepare.run_cpu_time_us":722,"prepare.run_wall_time_us":8555,"raft.queue_time_us":146,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":2705,"spinlock_wait_cycles":21176064,"thread_start_us":371472,"threads_started":1}
I20260321 02:59:38.519280 11385 raft_consensus.cc:1275] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260321 02:59:38.519125 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.519331 11379 raft_consensus.cc:2393] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 5: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:38.484282 14019 raft_consensus.cc:3060] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:38.496884 13950 raft_consensus.cc:2749] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:38.520354 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:38.496608 13873 consensus_queue.cc:1059] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260321 02:59:38.534238 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:38.514469 11306 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:38.537483 13567 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:38.482564 11381 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:38.544790 11229 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:38.509196 13920 raft_consensus.cc:2804] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:38.447781 14027 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:38.552000 14120 log.cc:927] Time spent T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.075s user 0.000s sys 0.001s
I20260321 02:59:38.533308 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.549626 11380 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:38.554198 11376 raft_consensus.cc:1240] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260321 02:59:38.554334 11942 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:38.565289 11349 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:38.570094 13920 raft_consensus.cc:697] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:38.571110 13920 consensus_queue.cc:237] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:38.577126 11229 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 8.
I20260321 02:59:38.577590 14027 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:38.578258 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:38.578305 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.579591 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:38.556986 11382 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:38.561194 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:38.562160 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.580893 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:38.501605 14089 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:38.511287 14024 raft_consensus.cc:2749] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:38.590538 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:38.591747 11380 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:38.580981 11942 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:38.595008 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:38.595942 11381 raft_consensus.cc:2393] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 8: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:38.441617 14098 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:38.598093 14074 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20260321 02:59:38.597929 14043 raft_consensus.cc:2804] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:38.598702 14043 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:38.599094 14043 raft_consensus.cc:3060] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:38.599256 14098 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:38.579694 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.600028 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.597977 11234 raft_consensus.cc:1240] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:38.527377 14021 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:38.581683 11229 raft_consensus.cc:2376] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 7. Re-sending same reply.
I20260321 02:59:38.583065 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.583525 11230 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:38.556393 11384 raft_consensus.cc:2410] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260321 02:59:38.588928 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:38.608253 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:38.608139 11382 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:38.608124 11349 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:38.610488 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:38.611415 11381 raft_consensus.cc:2393] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 10: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:38.611810 11376 raft_consensus.cc:2393] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 7: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:38.563977 11200 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:38.613708 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:38.614468 14118 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:38.615388 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:38.589455 14119 raft_consensus.cc:2804] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260321 02:59:38.633997 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:38.638165 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 74302us
I20260321 02:59:38.644486 11200 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:38.645694 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:38.646801 11200 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:38.506206 14121 raft_consensus.cc:2764] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260321 02:59:38.649555 11200 leader_election.cc:400] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Vote denied by peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267) with higher term. Message: Invalid argument: T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8.
I20260321 02:59:38.651904 11200 leader_election.cc:403] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260321 02:59:38.653920 11200 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:38.655616 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 98128.3us
I20260321 02:59:38.655982 11231 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:38.648769 13664 raft_consensus.cc:2749] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260321 02:59:38.651827 14027 raft_consensus.cc:2804] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:38.665249 14027 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:38.665691 14027 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:38.648509 14107 raft_consensus.cc:2749] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260321 02:59:38.668105 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:38.670154 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:38.672258 13970 consensus_queue.cc:1059] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260321 02:59:38.675022 13865 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:38.635512 14021 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:38.637256 14091 raft_consensus.cc:2749] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260321 02:59:38.649412 14119 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:38.614279 14096 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:38.689811 11376 raft_consensus.cc:1240] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:38.696884 13950 raft_consensus.cc:3055] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:38.697326 14096 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:38.697840 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.698161 11227 raft_consensus.cc:2393] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 5: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:38.697367 13950 raft_consensus.cc:740] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:38.687774 13664 raft_consensus.cc:2749] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:38.699497 13950 consensus_queue.cc:260] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:38.700321 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.701089 11306 raft_consensus.cc:2393] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:38.698654 11310 raft_consensus.cc:2393] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 9: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:38.702702 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.702929 11380 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:38.703313 11302 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:38.703588 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.704614 11306 raft_consensus.cc:2410] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 10 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260321 02:59:38.608512 14072 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:38.705838 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.707226 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.710381 13950 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:38.710667 14098 consensus_queue.cc:1048] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:59:38.711429 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.687683 14119 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:38.712225 11304 raft_consensus.cc:2393] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 8: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:38.713038 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:38.658264 14036 raft_consensus.cc:2749] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267) with higher term. Message: Invalid argument: T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8.
I20260321 02:59:38.714694 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.710309 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.715914 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.716638 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.717254 11302 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:38.718035 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.718539 11302 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:38.703616 11381 raft_consensus.cc:1240] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7. Ops: [5.3-6.4]
I20260321 02:59:38.710923 11309 raft_consensus.cc:1240] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260321 02:59:38.719254 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.719344 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.719664 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.720409 11309 raft_consensus.cc:2393] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:38.720219 14036 consensus_queue.cc:1048] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:59:38.711464 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:38.721700 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.722395 11303 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:38.724939 11304 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 7.
I20260321 02:59:38.724664 14020 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:38.734510 11200 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:38.735215 14020 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:38.725633 14099 consensus_queue.cc:1048] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:59:38.726552 14043 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:38.737856 14043 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:38.710278 11310 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:38.708741 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.728188 13718 consensus_queue.cc:1048] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:59:38.740195 14098 raft_consensus.cc:2804] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:38.741742 11306 raft_consensus.cc:2393] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 8: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:38.729128 11275 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:38.730080 13664 consensus_queue.cc:1048] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:59:38.738689 11980 raft_consensus.cc:2368] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6.
I20260321 02:59:38.741146 14098 raft_consensus.cc:697] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:38.743914 11231 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58214 (request call id 1356) took 1239 ms. Trace:
I20260321 02:59:38.727067 14021 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:38.753443 14072 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:38.754441 11382 raft_consensus.cc:1240] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:38.751212 11275 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:38.750459 14020 raft_consensus.cc:2804] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:38.750995 11231 rpcz_store.cc:276] 0321 02:59:37.504646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:37.504987 (+ 341us) service_pool.cc:225] Handling call
0321 02:59:38.743625 (+1238638us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540027f6d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:38.743743 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2458796288}
I20260321 02:59:38.746832 14098 consensus_queue.cc:237] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:38.756783 14020 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:38.757427 14020 raft_consensus.cc:3060] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:38.761672 13865 raft_consensus.cc:2764] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260321 02:59:38.761698 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:38.763147 11305 raft_consensus.cc:2376] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 8. Re-sending same reply.
I20260321 02:59:38.708582 11236 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:38.769384 11275 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:38.776801 11311 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
I20260321 02:59:38.807322 11275 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:38.780458 11302 raft_consensus.cc:1240] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:38.785027 11306 raft_consensus.cc:1275] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260321 02:59:38.809652 11275 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:38.810029 11306 pending_rounds.cc:85] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da: Aborting all ops after (but not including) 5
I20260321 02:59:38.810289 11306 pending_rounds.cc:107] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260321 02:59:38.786811 11305 raft_consensus.cc:1240] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:38.787592 13593 raft_consensus.cc:3055] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:38.787774 11304 raft_consensus.cc:1275] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260321 02:59:38.814771 13593 raft_consensus.cc:740] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:38.790923 11307 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 8.
I20260321 02:59:38.815737 13593 consensus_queue.cc:260] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:38.817793 13593 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:38.793607 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:38.818845 11309 raft_consensus.cc:2468] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 4.
I20260321 02:59:38.794028 11310 raft_consensus.cc:1240] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:38.801015 11303 raft_consensus.cc:1275] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20260321 02:59:38.820864 11303 pending_rounds.cc:85] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da: Aborting all ops after (but not including) 5
I20260321 02:59:38.750236 14027 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:38.821208 11303 pending_rounds.cc:107] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20260321 02:59:38.821498 11303 raft_consensus.cc:2889] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:38.822856 14027 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:38.812155 11306 raft_consensus.cc:2889] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:38.813923 14104 raft_consensus.cc:2804] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:38.814098 14021 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:38.791863 11201 leader_election.cc:400] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Vote denied by peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285) with higher term. Message: Invalid argument: T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6.
I20260321 02:59:38.776402 14072 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:38.820072 11236 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58214 (request call id 1364) took 1099 ms. Trace:
I20260321 02:59:38.849699 11236 rpcz_store.cc:276] 0321 02:59:37.721092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:37.721241 (+ 149us) service_pool.cc:225] Handling call
0321 02:59:38.819853 (+1098612us) spinlock_profiling.cc:243] Waited 954 ms on lock 0x7b540019d2d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:38.820000 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2193884928}
I20260321 02:59:38.832723 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.827184 11275 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:38.852465 11231 raft_consensus.cc:2393] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 8: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:38.853289 11275 leader_election.cc:400] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Vote denied by peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533) with higher term. Message: Invalid argument: T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7.
I20260321 02:59:38.855000 11275 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:38.837013 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.838538 11201 leader_election.cc:403] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20260321 02:59:38.785813 11348 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:38.843160 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:38.857905 11234 raft_consensus.cc:2376] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 8. Re-sending same reply.
I20260321 02:59:38.843350 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.843927 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:38.823755 11233 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:38.862205 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:38.862679 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:38.863025 11381 raft_consensus.cc:1275] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260321 02:59:38.863322 11382 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:38.865804 11275 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:38.844400 13933 raft_consensus.cc:2804] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:38.890682 13933 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:38.891105 13933 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:38.892172 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:38.892422 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:38.893074 11377 raft_consensus.cc:2393] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 5: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:38.895043 11380 raft_consensus.cc:1275] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260321 02:59:38.896934 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 105398us
W20260321 02:59:38.897059 11233 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58214 (request call id 1358) took 1181 ms (client timeout 1567 ms). Trace:
W20260321 02:59:38.897820 11233 rpcz_store.cc:269] 0321 02:59:37.715795 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:37.715922 (+ 127us) service_pool.cc:225] Handling call
0321 02:59:38.896815 (+1180893us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x7b54002f94d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:38.896984 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2276427008}
I20260321 02:59:38.898368 14027 raft_consensus.cc:3060] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:38.831910 14118 raft_consensus.cc:2749] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:38.827414 13950 raft_consensus.cc:2804] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:38.838037 14104 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:38.911029 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 45399.7us
I20260321 02:59:38.892861 11376 raft_consensus.cc:2468] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 3.
I20260321 02:59:38.914897 11275 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:38.844216 11200 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:38.903204 11382 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:38.862002 13865 raft_consensus.cc:2749] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260321 02:59:38.916733 11275 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:38.917873 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 73732.7us
I20260321 02:59:38.920253 11200 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:38.904740 14019 raft_consensus.cc:2804] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:38.923336 13920 raft_consensus.cc:2749] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260321 02:59:38.939980 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 346190us
I20260321 02:59:38.864005 14072 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:38.924975 14089 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:38.936674 14099 consensus_queue.cc:1048] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:59:38.926146 13664 raft_consensus.cc:2749] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:38.937721 14074 raft_consensus.cc:2804] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:38.860311 13733 consensus_queue.cc:1048] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.061s
I20260321 02:59:38.909327 13950 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:38.914023 14104 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:38.856899 14021 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:38.922544 13567 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election won for term 9
I20260321 02:59:38.928965 11275 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:38.944624 14019 raft_consensus.cc:697] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:38.950985 14074 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:38.984845 14074 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:38.985857 14090 consensus_queue.cc:1048] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260321 02:59:38.965059 11275 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:38.983471 13733 consensus_queue.cc:1059] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:59:38.983301 14019 consensus_queue.cc:237] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:38.984282 13865 raft_consensus.cc:2749] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260321 02:59:38.954483 13950 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:39.002902 11349 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:39.004137 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:39.009159 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:39.017091 11232 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:38.989607 13567 raft_consensus.cc:697] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 9 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:39.022132 11200 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:39.018623 11349 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:39.024894 11349 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:39.026356 11349 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:39.008771 14024 raft_consensus.cc:2804] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:39.029605 11200 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:38.998694 14099 consensus_queue.cc:1059] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260321 02:59:39.025533 14118 raft_consensus.cc:2804] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:39.031279 11349 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:39.051167 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:39.068594 11382 raft_consensus.cc:2393] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 7: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:39.070343 11380 raft_consensus.cc:1275] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260321 02:59:39.055748 11227 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:39.071935 11227 raft_consensus.cc:606] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:39.072402 11227 raft_consensus.cc:614] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:39.052177 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:39.037204 14036 raft_consensus.cc:2804] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260321 02:59:39.033334 11275 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:39.082762 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 49783.3us
I20260321 02:59:39.029299 13865 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:39.008411 14115 consensus_queue.cc:1059] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260321 02:59:39.021904 14021 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:39.073361 11311 raft_consensus.cc:2368] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8.
I20260321 02:59:39.037346 14091 raft_consensus.cc:2749] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260321 02:59:39.044616 13733 raft_consensus.cc:2804] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:39.011420 13970 consensus_queue.cc:1048] T 5aaa96ba673046439e87b58b0e693526 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.035s
I20260321 02:59:39.084318 11275 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:39.063402 11232 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
W20260321 02:59:39.085058 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 54024.5us
I20260321 02:59:39.118120 11349 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:39.087463 11376 raft_consensus.cc:1275] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260321 02:59:39.088470 14117 raft_consensus.cc:2749] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:39.088948 14122 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:39.092777 11305 raft_consensus.cc:1275] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260321 02:59:39.123407 11305 pending_rounds.cc:85] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da: Aborting all ops after (but not including) 4
I20260321 02:59:39.123747 11305 pending_rounds.cc:107] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260321 02:59:39.124111 11305 raft_consensus.cc:2889] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260321 02:59:39.124526 14131 log.cc:927] Time spent T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20260321 02:59:39.124740 13933 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:39.126066 14133 log.cc:927] Time spent T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
I20260321 02:59:39.126982 13933 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:39.128248 14089 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:39.129016 14072 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:39.129635 13567 consensus_queue.cc:237] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.092886 14099 raft_consensus.cc:3055] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:39.132244 14099 raft_consensus.cc:740] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:39.133107 14099 consensus_queue.cc:260] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.134320 14099 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:39.094221 13718 raft_consensus.cc:3055] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Stepping down as leader of term 8
I20260321 02:59:39.135622 13718 raft_consensus.cc:740] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:39.135200 14119 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:39.136535 13718 consensus_queue.cc:260] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.137709 13718 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:39.137776 14119 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:39.025198 14106 consensus_queue.cc:1048] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.070s
I20260321 02:59:39.097718 14098 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:39.139874 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:39.140508 14098 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:39.142478 14098 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:39.098333 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:39.036832 14121 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:39.044056 11132 catalog_manager.cc:5671] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 3 to 4, leader changed from 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 4 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
W20260321 02:59:39.146845 14132 log.cc:927] Time spent T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.138s user 0.001s sys 0.000s
I20260321 02:59:39.105141 11311 raft_consensus.cc:1275] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260321 02:59:39.029078 14076 raft_consensus.cc:2749] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:39.113722 14107 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:39.057238 14024 raft_consensus.cc:697] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:39.082018 14036 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:39.067487 14118 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:39.119475 11349 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:39.148067 11976 raft_consensus.cc:1240] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:39.156059 11232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:39.160852 11232 raft_consensus.cc:606] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:39.162083 11132 catalog_manager.cc:5671] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 3 to 7, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 7 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
W20260321 02:59:39.171252 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 51841.7us
W20260321 02:59:39.173977 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 233106us
I20260321 02:59:39.150170 14091 raft_consensus.cc:2749] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:39.172312 11349 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:39.177004 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:39.179047 11980 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:39.167215 11132 catalog_manager.cc:5671] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:59:39.121819 13733 raft_consensus.cc:697] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:39.148507 14107 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:39.179255 11349 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:39.180756 11132 catalog_manager.cc:5671] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:59:39.181998 11201 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:39.188449 11349 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:39.188787 11132 catalog_manager.cc:5671] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 2 to 6, leader changed from 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 6 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:59:39.190409 11132 catalog_manager.cc:5671] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:59:39.191668 11132 catalog_manager.cc:5671] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:59:39.194689 11349 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:39.159235 14036 raft_consensus.cc:3060] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:39.197948 11385 raft_consensus.cc:1240] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:39.194029 11132 catalog_manager.cc:5671] T 709aaa09afd74a25bbd5dc62dca37f02 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 4 to 7, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4). New cstate: current_term: 7 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:59:39.264374 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:39.457770 11384 raft_consensus.cc:3060] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:39.255915 11134 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42514 (request call id 211) took 9716 ms. Trace:
I20260321 02:59:39.445313 11232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:39.461258 11232 raft_consensus.cc:606] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 LEADER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:39.449575 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 365096us
I20260321 02:59:39.465083 11275 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:39.452472 11310 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.629s user 0.011s sys 0.018s
I20260321 02:59:39.256723 11350 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
W20260321 02:59:39.466854 11310 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s user 0.011s sys 0.018s
W20260321 02:59:39.469063 11350 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 212274us
I20260321 02:59:39.470911 11232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
W20260321 02:59:39.348479 11942 leader_election.cc:336] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Timed out: RequestConsensusVote RPC to 127.9.213.1:36717 timed out after 1.604s (SENT)
W20260321 02:59:39.258179 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 76326us
W20260321 02:59:39.473713 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 125108us
I20260321 02:59:39.262118 13664 raft_consensus.cc:2804] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260321 02:59:39.475123 11942 leader_election.cc:336] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.604s (SENT)
I20260321 02:59:39.475179 13664 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:39.460116 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:39.475905 13664 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:39.475785 11942 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:39.477293 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:39.478644 11942 connection.cc:232] RPC call timeout handler was delayed by -1.1737448879999874s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3285s for a response
W20260321 02:59:39.387347 13593 consensus_meta.cc:287] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.569s user 0.013s sys 0.001s
I20260321 02:59:39.199141 11349 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:39.487565 13723 consensus_queue.cc:1048] T 48b15e4599b848c8a833e37849eabbda P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:59:39.490181 13723 consensus_queue.cc:1048] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:59:39.114435 13865 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:39.364949 13733 consensus_queue.cc:237] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:39.495206 13865 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:39.459200 11132 catalog_manager.cc:5671] T 2b701d6135bc458c9aef1d5303344525 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:59:39.496456 13846 raft_consensus.cc:2804] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260321 02:59:39.186877 14020 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.175504 13873 raft_consensus.cc:2749] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:39.498092 14043 raft_consensus.cc:2749] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:39.175197 14021 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:39.154480 14074 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.500977 14020 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:39.501183 11233 raft_consensus.cc:1240] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260321 02:59:39.354727 14104 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.254583 14072 raft_consensus.cc:2749] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:39.198697 14076 raft_consensus.cc:2804] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:39.399665 14107 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:39.259809 14089 raft_consensus.cc:2749] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:39.284165 14024 consensus_queue.cc:237] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.496994 13865 raft_consensus.cc:2749] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260321 02:59:39.198369 14117 raft_consensus.cc:2804] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:39.547351 14117 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:39.548067 14117 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:39.164508 14118 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:39.548393 14076 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:39.549054 14074 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:39.549288 14076 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
W20260321 02:59:39.368814 14138 log.cc:927] Time spent T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.103s user 0.000s sys 0.001s
W20260321 02:59:39.405880 14143 log.cc:927] Time spent T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c: Append to log took a long time: real 0.278s user 0.004s sys 0.000s
I20260321 02:59:39.556788 11308 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1777) took 1097 ms. Trace:
I20260321 02:59:39.407760 14122 raft_consensus.cc:2749] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:39.485574 11275 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:39.460435 11134 rpcz_store.cc:276] 0321 02:59:29.540130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:29.540739 (+ 609us) service_pool.cc:225] Handling call
0321 02:59:39.208855 (+9668116us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:59:35.377074 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:59:35.493729 (+116655us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:59:35.493769 (+ 40us) write_op.cc:435] Acquired schema lock
0321 02:59:35.493785 (+ 16us) tablet.cc:661] Decoding operations
0321 02:59:35.600022 (+106237us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:59:35.605726 (+ 5704us) write_op.cc:641] Partition lock acquired for write op
0321 02:59:35.611780 (+ 6054us) tablet.cc:684] Acquiring locks for 11 operations
0321 02:59:36.108162 (+496382us) tablet.cc:700] Row locks acquired
0321 02:59:36.108186 (+ 24us) write_op.cc:260] PREPARE: finished
0321 02:59:36.108345 (+ 159us) write_op.cc:270] Start()
0321 02:59:36.108472 (+ 127us) write_op.cc:276] Timestamp: P: 1774061976108294 usec, L: 0
0321 02:59:36.108491 (+ 19us) op_driver.cc:348] REPLICATION: starting
0321 02:59:36.120260 (+ 11769us) log.cc:844] Serialized 4346 byte log entry
0321 02:59:37.860039 (+1739779us) op_driver.cc:464] REPLICATION: finished
0321 02:59:38.335426 (+475387us) spinlock_profiling.cc:243] Waited 7.08 ms on lock 0x7b440002304c. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93c2dc5 00007fa8a93c37e2 00007fa8ac0f757b 00007fa8ac0f2e4f 00007fa8a9498fbc 00007fa8b2276b19 00007fa8a94b546b 00007fa8a94b370f 00007fa8a94b59d2 00007fa8aff8d810
0321 02:59:38.548079 (+212653us) write_op.cc:301] APPLY: starting
0321 02:59:38.548215 (+ 136us) tablet.cc:1366] starting BulkCheckPresence
0321 02:59:38.548482 (+ 267us) tablet.cc:1369] finished BulkCheckPresence
0321 02:59:38.548504 (+ 22us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:59:38.764076 (+215572us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:59:38.764309 (+ 233us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0321 02:59:38.764352 (+ 43us) write_op.cc:312] APPLY: finished
0321 02:59:38.772605 (+ 8253us) log.cc:844] Serialized 83 byte log entry
0321 02:59:38.947713 (+175108us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:38.953143 (+ 5430us) write_op.cc:454] Released schema lock
0321 02:59:38.955751 (+ 2608us) write_op.cc:341] FINISH: Updating metrics
0321 02:59:38.956216 (+ 465us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:38.956258 (+ 42us) write_op.cc:454] Released schema lock
0321 02:59:39.397595 (+441337us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b440002304c. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93c2dc5 00007fa8a93c37e2 00007fa8ac0f757b 00007fa8ac0f2e4f 00007fa8ac0e6c0e 00007fa8ac0e5f56 00007fa8ac0e5b78 00007fa8ac0ef5c1 00007fa8ac0ef579 00007fa8ac0ef509
Metrics: {"mutex_wait_us":556880,"reported_tablets":40,"child_traces":[["op",{"apply.queue_time_us":625953,"apply.run_cpu_time_us":36902,"apply.run_wall_time_us":438065,"num_ops":11,"prepare.queue_time_us":225062,"prepare.run_cpu_time_us":23423,"prepare.run_wall_time_us":1860510,"raft.queue_time_us":477871,"raft.run_cpu_time_us":17413,"raft.run_wall_time_us":776255,"replication_time_us":1698722,"spinlock_wait_cycles":67452032,"thread_start_us":2375962,"threads_started":5,"wal-append.queue_time_us":1484298,"wal-append.run_cpu_time_us":14763,"wal-append.run_wall_time_us":1490075}]]}
I20260321 02:59:39.502573 11384 raft_consensus.cc:2468] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 6.
I20260321 02:59:39.507527 11304 raft_consensus.cc:1240] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260321 02:59:39.508299 14153 raft_consensus.cc:2749] T 709aaa09afd74a25bbd5dc62dca37f02 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:39.560540 13593 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:39.567523 14074 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:39.572489 13593 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:39.573423 14074 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.576256 13723 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:39.577129 13723 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:59:39.579883 14021 raft_consensus.cc:2749] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260321 02:59:39.581575 13723 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:39.586429 11309 raft_consensus.cc:1275] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:39.521097 14163 raft_consensus.cc:2804] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election won for term 7
W20260321 02:59:39.521262 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 322066us
I20260321 02:59:39.512192 14091 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:39.533672 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 54319.7us
I20260321 02:59:39.536808 14164 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:39.538086 14104 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:39.530510 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:39.558651 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 48916.5us
I20260321 02:59:39.557284 11308 rpcz_store.cc:276] 0321 02:59:38.458884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:38.459026 (+ 142us) service_pool.cc:225] Handling call
0321 02:59:38.498124 (+ 39098us) raft_consensus.cc:1471] Updating replica for 2 ops
0321 02:59:38.566365 (+ 68241us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:38.566416 (+ 51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0321 02:59:38.580613 (+ 14197us) log.cc:844] Serialized 52 byte log entry
0321 02:59:39.011413 (+430800us) raft_consensus.cc:1645] Marking committed up to 7
0321 02:59:39.345907 (+334494us) log.cc:844] Serialized 14 byte log entry
0321 02:59:39.450089 (+104182us) log.cc:844] Serialized 14 byte log entry
0321 02:59:39.450955 (+ 866us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:39.452941 (+ 1986us) spinlock_profiling.cc:243] Waited 65.6 ms on lock 0x7b540035cad4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:39.453025 (+ 84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:39.556319 (+103294us) raft_consensus.cc:1685] finished
0321 02:59:39.556346 (+ 27us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:39.556547 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":104582,"prepare.queue_time_us":272398,"prepare.run_cpu_time_us":238,"prepare.run_wall_time_us":3675,"raft.queue_time_us":49530,"raft.run_cpu_time_us":600,"raft.run_wall_time_us":11681,"spinlock_wait_cycles":150867072,"thread_start_us":735306,"threads_started":2,"wal-append.queue_time_us":564044}
I20260321 02:59:39.561177 11132 catalog_manager.cc:5671] T 515945e5685c4afe9d41d5317acb434c P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:59:39.575764 14074 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:39.593329 11980 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:39.598752 14091 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.602252 14091 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:39.603210 14104 raft_consensus.cc:2749] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:39.596565 11349 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:39.608947 14121 raft_consensus.cc:2749] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:39.611089 11276 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:39.611840 13846 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:39.612428 13846 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:39.613775 13664 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:39.615700 14104 raft_consensus.cc:2804] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:39.616178 14104 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:39.616578 14104 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:39.622807 14122 consensus_queue.cc:1059] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260321 02:59:39.624162 13865 raft_consensus.cc:3055] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:39.624588 13865 raft_consensus.cc:740] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:39.624747 14074 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:39.625653 14074 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:39.626020 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:39.626636 11976 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:39.626505 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:39.625047 13593 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:39.624733 13664 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:39.632858 11200 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:39.643026 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:39.644629 13920 raft_consensus.cc:3055] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:39.645112 13920 raft_consensus.cc:740] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:39.646500 13920 consensus_queue.cc:260] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.627894 14074 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:39.648562 11349 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:39.622925 11132 catalog_manager.cc:5671] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:59:39.628234 14163 raft_consensus.cc:697] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:39.657159 14074 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election won for term 9
I20260321 02:59:39.636948 13865 consensus_queue.cc:260] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:39.657980 14074 raft_consensus.cc:697] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:39.659433 13865 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:39.659848 14074 consensus_queue.cc:237] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 8.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.662027 14021 raft_consensus.cc:2804] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:39.665419 14021 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:39.665823 14021 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:39.668980 14076 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.671769 14117 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.678031 14118 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.672770 14163 consensus_queue.cc:237] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:39.651296 14121 raft_consensus.cc:2804] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:39.680214 14121 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:39.680729 14121 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
W20260321 02:59:39.681972 13950 consensus_meta.cc:287] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.661s user 0.000s sys 0.018s
I20260321 02:59:39.682418 13950 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.683810 11307 raft_consensus.cc:1240] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260321 02:59:39.684458 14117 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:39.678081 14021 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:39.687418 13950 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:39.688288 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 202949us
I20260321 02:59:39.688443 14164 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.674467 11976 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:39.690549 11275 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:39.656416 11942 leader_election.cc:336] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Timed out: RequestConsensusVote RPC to 127.9.213.1:36717 timed out after 1.997s (SENT)
I20260321 02:59:39.662868 11349 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:39.696360 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 62325.1us
I20260321 02:59:39.690929 14164 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:39.694798 14021 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:39.701009 14163 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:39.641927 13846 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.705648 13920 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:39.705745 14024 raft_consensus.cc:2749] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:39.706549 11275 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:39.708843 11942 leader_election.cc:336] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.997s (SENT)
I20260321 02:59:39.709285 11942 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:39.707599 11201 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
W20260321 02:59:39.710237 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 77690.7us
I20260321 02:59:39.711778 13723 raft_consensus.cc:2804] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:39.712252 11200 leader_election.cc:400] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Vote denied by peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267) with higher term. Message: Invalid argument: T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8.
I20260321 02:59:39.712342 13723 raft_consensus.cc:697] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:39.712638 11200 leader_election.cc:403] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260321 02:59:39.713495 13733 consensus_queue.cc:1048] T ca80e3e3473540618f654536751cdc3b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:59:39.713441 13723 consensus_queue.cc:237] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:39.715688 13593 raft_consensus.cc:2749] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267) with higher term. Message: Invalid argument: T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8.
I20260321 02:59:39.716313 14104 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:39.717459 14119 raft_consensus.cc:2749] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:39.718382 11349 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:39.718581 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:39.719240 14036 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.720842 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:39.721884 11976 raft_consensus.cc:2393] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 8: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:39.722676 11232 raft_consensus.cc:606] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:39.723124 11232 raft_consensus.cc:614] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:39.725443 13593 consensus_queue.cc:1048] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:59:39.726397 13664 consensus_queue.cc:1048] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.000s
I20260321 02:59:39.711109 13846 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:39.729547 11232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:39.730316 11232 raft_consensus.cc:606] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:39.728755 14036 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:39.739506 14024 raft_consensus.cc:2749] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:39.740957 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:39.740684 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:39.740306 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:39.741791 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260321 02:59:39.742851 11349 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:39.744988 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:39.745834 14099 consensus_meta.cc:287] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.611s user 0.009s sys 0.009s
I20260321 02:59:39.745922 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:39.746136 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:39.744974 14164 raft_consensus.cc:2804] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:39.705276 14118 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:39.746944 11384 raft_consensus.cc:2376] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7. Re-sending same reply.
I20260321 02:59:39.747097 14164 raft_consensus.cc:697] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:39.751633 14117 raft_consensus.cc:2804] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:39.752357 14117 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
W20260321 02:59:39.752466 13718 consensus_meta.cc:287] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.614s user 0.012s sys 0.010s
I20260321 02:59:39.752988 14117 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
W20260321 02:59:39.756572 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 50154.9us
I20260321 02:59:39.756844 14162 consensus_queue.cc:1048] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:59:39.757254 14036 consensus_queue.cc:1059] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:59:39.757726 11275 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:39.759177 13733 consensus_queue.cc:1059] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:59:39.760176 13865 raft_consensus.cc:2804] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:39.760426 14151 raft_consensus.cc:3055] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:39.761054 14151 raft_consensus.cc:740] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:39.761505 13865 raft_consensus.cc:697] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:39.762209 14162 raft_consensus.cc:3055] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:39.762347 14151 consensus_queue.cc:260] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.764163 14162 raft_consensus.cc:740] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
W20260321 02:59:39.763396 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:39.765018 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:59:39.764691 14151 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:39.766285 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:39.718618 14104 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:39.768111 14027 consensus_meta.cc:287] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.842s user 0.005s sys 0.017s
I20260321 02:59:39.762779 13865 consensus_queue.cc:237] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.765800 14162 consensus_queue.cc:260] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.759681 11275 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:39.778602 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:39.780156 11232 raft_consensus.cc:2410] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 8 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260321 02:59:39.780330 11275 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:39.781450 14104 raft_consensus.cc:2804] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:39.782064 14104 raft_consensus.cc:697] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:39.784340 13933 raft_consensus.cc:2804] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:39.784940 13933 raft_consensus.cc:697] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:39.785909 13933 consensus_queue.cc:237] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.783252 14104 consensus_queue.cc:237] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.787647 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:39.788568 11381 raft_consensus.cc:2410] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260321 02:59:39.789100 11275 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:39.790091 13846 raft_consensus.cc:2804] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:39.790298 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:39.790562 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:39.791018 11377 raft_consensus.cc:2376] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7. Re-sending same reply.
I20260321 02:59:39.791110 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:39.791266 11383 raft_consensus.cc:2376] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6. Re-sending same reply.
I20260321 02:59:39.754083 14164 consensus_queue.cc:237] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:39.791707 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:39.791864 11379 raft_consensus.cc:2376] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7. Re-sending same reply.
I20260321 02:59:39.792479 11381 raft_consensus.cc:2368] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 7.
I20260321 02:59:39.793524 11378 raft_consensus.cc:1275] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260321 02:59:39.727586 14121 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:39.793404 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:39.742805 14076 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:39.794338 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:39.768743 14027 raft_consensus.cc:2749] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285) with higher term. Message: Invalid argument: T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6.
I20260321 02:59:39.804242 11385 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:39.800019 13846 raft_consensus.cc:697] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:39.813115 14162 raft_consensus.cc:3060] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:39.814437 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:39.814735 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:39.821529 11307 raft_consensus.cc:2393] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:39.823843 11304 raft_consensus.cc:2393] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 9: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:39.824199 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:39.827275 11381 raft_consensus.cc:2376] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5. Re-sending same reply.
I20260321 02:59:39.825620 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:39.824527 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:39.823976 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:39.824815 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:39.843855 11385 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:39.850374 14076 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:39.851292 11942 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:39.851046 14076 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:39.852936 11942 leader_election.cc:400] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267) with higher term. Message: Invalid argument: T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 5. Current term is 6.
I20260321 02:59:39.853144 14119 consensus_queue.cc:1059] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260321 02:59:39.854156 14119 raft_consensus.cc:3055] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:39.854496 14119 raft_consensus.cc:740] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:39.855490 11942 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:39.852942 14090 raft_consensus.cc:2764] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260321 02:59:39.808467 13920 consensus_queue.cc:1059] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.026s
I20260321 02:59:39.881862 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:39.884423 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:39.848371 13846 consensus_queue.cc:237] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.886185 11349 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:39.872013 14118 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:39.878793 14121 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:39.878453 14117 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.885912 13920 raft_consensus.cc:3055] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:39.890388 14090 raft_consensus.cc:2749] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:39.898777 14076 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:39.855535 14119 consensus_queue.cc:260] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:39.907310 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 51927us
I20260321 02:59:39.895496 13920 raft_consensus.cc:740] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:39.897871 14164 raft_consensus.cc:2749] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260321 02:59:39.908429 13950 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:39.967264 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 86625.8us
I20260321 02:59:39.920158 14118 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:39.969334 11275 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:39.980480 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:39.981244 11232 raft_consensus.cc:1275] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260321 02:59:39.982883 14117 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:39.983459 11304 raft_consensus.cc:1275] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260321 02:59:39.984396 14118 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:39.985721 11383 raft_consensus.cc:1275] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260321 02:59:39.992606 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:40.005064 14119 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:40.004331 13933 raft_consensus.cc:2749] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:40.009969 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:40.011091 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 60199.4us
I20260321 02:59:39.968369 13950 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:40.020079 13950 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:40.012904 13920 consensus_queue.cc:260] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:40.021775 11378 raft_consensus.cc:1275] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:40.021972 11304 raft_consensus.cc:1275] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260321 02:59:40.022292 11304 pending_rounds.cc:85] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da: Aborting all ops after (but not including) 4
I20260321 02:59:40.022550 11304 pending_rounds.cc:107] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260321 02:59:40.022827 11304 raft_consensus.cc:2889] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:40.025734 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:40.026755 11307 raft_consensus.cc:2393] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 8: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:40.014151 14163 raft_consensus.cc:2749] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:40.038666 11379 raft_consensus.cc:1275] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:40.040936 11378 raft_consensus.cc:1240] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260321 02:59:40.044611 11232 raft_consensus.cc:2410] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260321 02:59:40.048599 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:40.048852 11309 raft_consensus.cc:2468] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:40.049629 11227 raft_consensus.cc:2410] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20260321 02:59:40.013794 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 47698.7us
I20260321 02:59:40.056763 11384 raft_consensus.cc:1275] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260321 02:59:40.060467 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:40.057322 11384 pending_rounds.cc:85] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5: Aborting all ops after (but not including) 4
I20260321 02:59:40.065188 11384 pending_rounds.cc:107] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260321 02:59:40.065443 11384 raft_consensus.cc:2889] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:40.066372 14163 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:40.066731 11304 raft_consensus.cc:1240] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:40.067016 14163 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:40.068029 14024 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:40.070365 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:40.071194 14118 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:40.072755 14118 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:40.076336 13920 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:40.075806 14118 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:40.084975 11275 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:40.085368 11304 raft_consensus.cc:1275] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:40.068571 14024 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:40.089175 14024 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:40.091185 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:40.091967 14163 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:40.094935 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:40.096033 11378 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:40.098706 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:40.101081 11310 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:40.101542 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:40.101680 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:40.102299 11305 raft_consensus.cc:3060] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:40.098960 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:40.103312 11303 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
I20260321 02:59:40.099895 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:40.104852 11306 raft_consensus.cc:2393] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 8: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:40.108363 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:40.108582 11232 raft_consensus.cc:1275] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260321 02:59:40.110067 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:40.110750 11305 raft_consensus.cc:2468] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:40.110884 11379 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
I20260321 02:59:40.124342 11349 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:40.127943 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:40.128736 11385 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:40.129838 14107 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:40.130892 14024 raft_consensus.cc:2804] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:40.130563 14107 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:40.132501 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:40.134400 11379 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:40.135048 14162 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:40.135253 14024 raft_consensus.cc:697] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:40.135380 11350 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:40.139773 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:40.118845 14104 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:40.135398 14122 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:40.148423 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:40.152753 11381 raft_consensus.cc:1240] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:40.152652 14024 consensus_queue.cc:237] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:40.153880 14107 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:40.155774 14163 raft_consensus.cc:2749] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:40.156733 14163 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:40.170836 11379 raft_consensus.cc:3060] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Advancing to term 5
I20260321 02:59:40.157253 14163 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:40.169710 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 84817.8us
I20260321 02:59:40.143015 14118 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:40.141577 14162 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:40.176977 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 56392.4us
I20260321 02:59:40.177098 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:40.227597 11980 raft_consensus.cc:1275] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260321 02:59:40.230209 11384 raft_consensus.cc:1240] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260321 02:59:40.232856 11275 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:40.233995 11349 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:40.228189 11980 pending_rounds.cc:85] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56: Aborting all ops after (but not including) 7
I20260321 02:59:40.235984 11980 pending_rounds.cc:107] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260321 02:59:40.236268 11980 raft_consensus.cc:2889] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260321 02:59:40.242378 11350 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 107066us
I20260321 02:59:40.240553 13865 raft_consensus.cc:2749] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:40.246481 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:40.249714 14091 raft_consensus.cc:2749] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260321 02:59:40.253301 14163 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:40.254678 14162 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:40.254804 11349 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:40.182719 14104 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:40.267437 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:40.276751 11980 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:40.279501 11378 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:40.238469 14122 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:40.289927 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:40.290592 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 64807.9us
I20260321 02:59:40.287362 11201 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:40.264307 13664 consensus_queue.cc:1048] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20260321 02:59:40.299093 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:40.302135 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:40.282333 14163 raft_consensus.cc:2749] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:40.307835 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:40.325047 11381 raft_consensus.cc:1275] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260321 02:59:40.311439 14027 raft_consensus.cc:2804] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:40.304915 14162 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:40.298298 13723 consensus_queue.cc:1048] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260321 02:59:40.318666 14117 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:40.369895 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 126050us
I20260321 02:59:40.333631 14027 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:40.337116 14091 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:40.374660 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 63264.5us
I20260321 02:59:40.380693 11306 raft_consensus.cc:1240] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 9. Current term is 10. Ops: []
W20260321 02:59:40.377420 14185 log.cc:927] Time spent T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.110s user 0.001s sys 0.000s
W20260321 02:59:40.395665 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 82157.3us
I20260321 02:59:40.380040 14027 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:40.368655 14117 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:40.397900 14187 log.cc:927] Time spent T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.148s user 0.001s sys 0.001s
I20260321 02:59:40.417701 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:40.379200 14162 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:40.415431 14091 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:40.425798 14104 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:40.419045 13664 consensus_queue.cc:1048] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260321 02:59:40.452770 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:40.453132 13723 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:40.453660 13723 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:40.450688 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:40.455592 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 54419.2us
I20260321 02:59:40.474222 11304 raft_consensus.cc:2410] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260321 02:59:40.500089 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:40.439289 11940 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:40.464130 11232 raft_consensus.cc:2468] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 6.
I20260321 02:59:40.455711 13723 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:40.457085 14144 log.cc:927] Time spent T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
W20260321 02:59:40.459128 14150 log.cc:927] Time spent T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
W20260321 02:59:40.456374 14192 log.cc:927] Time spent T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.093s user 0.001s sys 0.000s
I20260321 02:59:40.565284 11980 raft_consensus.cc:2468] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
W20260321 02:59:40.456846 14186 log.cc:927] Time spent T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
W20260321 02:59:40.566395 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 93591.5us
I20260321 02:59:40.566758 11379 raft_consensus.cc:2468] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:40.572500 11275 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
W20260321 02:59:40.468469 14156 log.cc:927] Time spent T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.085s user 0.000s sys 0.000s
I20260321 02:59:40.470129 14162 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:40.577867 11276 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:40.471650 14117 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:40.459599 14076 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:40.447710 14119 consensus_queue.cc:1059] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.120s
I20260321 02:59:40.481300 14207 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:40.498564 14122 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:40.565132 13920 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:40.589529 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 163503us
I20260321 02:59:40.471247 14027 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:40.610742 14196 log.cc:927] Time spent T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.168s user 0.001s sys 0.000s
I20260321 02:59:40.602651 13865 raft_consensus.cc:2804] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election won for term 7
W20260321 02:59:40.612790 14197 log.cc:927] Time spent T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.183s user 0.000s sys 0.001s
I20260321 02:59:40.613015 13865 raft_consensus.cc:697] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:40.614008 13920 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:40.614126 13865 consensus_queue.cc:237] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:40.616154 13920 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:40.617808 14091 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:40.620251 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:40.623253 14091 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:40.625456 14027 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:40.625022 14091 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:40.600209 14024 raft_consensus.cc:2804] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:40.626931 14024 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:40.627285 14024 raft_consensus.cc:3060] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:40.602447 14207 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:40.602033 14076 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:40.634315 14091 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:40.623260 14118 raft_consensus.cc:2749] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:40.634838 14210 raft_consensus.cc:3055] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:40.635257 14210 raft_consensus.cc:740] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:40.635573 14207 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:40.635998 11376 raft_consensus.cc:2410] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260321 02:59:40.636431 14118 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:40.641861 14118 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:40.644169 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 78275.1us
I20260321 02:59:40.644564 14118 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:40.636094 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:40.660322 11974 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Advancing to term 7
I20260321 02:59:40.647724 14208 consensus_queue.cc:1048] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:40.664080 14122 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:40.664677 14122 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:40.648388 13723 consensus_queue.cc:1059] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:59:40.666072 13723 raft_consensus.cc:3055] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:40.666404 13723 raft_consensus.cc:740] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:40.666692 14122 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:40.667192 13723 consensus_queue.cc:260] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:40.648280 14209 consensus_queue.cc:1048] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:59:40.649943 14215 consensus_queue.cc:1048] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:59:40.645048 11275 leader_election.cc:400] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Vote denied by peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533) with higher term. Message: Invalid argument: T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 7.
I20260321 02:59:40.652442 11200 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:40.655869 14207 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:40.660184 11942 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:40.637413 14210 consensus_queue.cc:260] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:40.667850 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:40.686193 13723 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:40.687176 14208 raft_consensus.cc:2749] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260321 02:59:40.688752 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 43841.3us
I20260321 02:59:40.690052 11275 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:40.690822 14122 raft_consensus.cc:2749] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:40.690701 14200 raft_consensus.cc:2804] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:40.686075 14207 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:40.692059 14200 raft_consensus.cc:697] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:40.693935 14200 consensus_queue.cc:237] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:40.635430 14076 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:40.697639 11275 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
W20260321 02:59:40.691457 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 59912.1us
I20260321 02:59:40.698992 14122 raft_consensus.cc:2804] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:40.699543 14122 raft_consensus.cc:697] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:40.699829 14210 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:40.700496 14122 consensus_queue.cc:237] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:40.705296 11275 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:40.706096 14119 consensus_queue.cc:1059] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260321 02:59:40.708819 14144 log.cc:927] Time spent T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
I20260321 02:59:40.707446 14119 raft_consensus.cc:3055] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:40.709810 14119 raft_consensus.cc:740] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:40.710964 14119 consensus_queue.cc:260] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:40.719820 14119 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:40.721251 11974 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:40.712133 13846 raft_consensus.cc:2749] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:40.725605 11201 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
W20260321 02:59:40.703801 14204 log.cc:927] Time spent T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.074s user 0.003s sys 0.000s
I20260321 02:59:40.726621 11200 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:40.735546 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:40.739120 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:40.743525 14027 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260321 02:59:40.743996 14027 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:40.743263 11200 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
W20260321 02:59:40.767045 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 212031us
I20260321 02:59:40.744496 14027 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:40.736337 14215 raft_consensus.cc:2804] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:40.782167 11380 raft_consensus.cc:1240] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:40.738040 14024 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:40.742517 14074 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:40.783471 11973 raft_consensus.cc:1275] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260321 02:59:40.796288 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 91071.5us
I20260321 02:59:40.789260 13933 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:40.800822 14074 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:40.825244 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:40.797399 14209 raft_consensus.cc:2804] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:40.793200 14215 raft_consensus.cc:697] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:40.796193 14162 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:40.833165 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 44528.5us
W20260321 02:59:40.839684 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 96511.9us
I20260321 02:59:40.840691 11200 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:40.829312 14209 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:40.846616 11132 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44688 (request call id 131) took 2904 ms. Trace:
I20260321 02:59:40.847530 11200 leader_election.cc:304] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:40.831859 14207 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:40.814195 13933 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:40.852102 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 43829.4us
I20260321 02:59:40.848907 11200 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:40.846122 13664 raft_consensus.cc:2804] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:40.847766 11132 rpcz_store.cc:276] 0321 02:59:37.939968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:37.993985 (+ 54017us) service_pool.cc:225] Handling call
0321 02:59:38.572498 (+578513us) spinlock_profiling.cc:243] Waited 484 ms on lock 0x7b54002d69d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b6470 00007fa8ac55531a 00007fa8b22e365c 00007fa8b23dd5de 00007fa8b08d64a4 00007fa8b08d6432 00007fa8b08d6361 00007fa8b08d62dc 00007fa8b08d5592 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:40.843802 (+2271304us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:59:39.817304 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:59:39.908684 (+ 91380us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:59:39.927191 (+ 18507us) write_op.cc:435] Acquired schema lock
0321 02:59:39.928628 (+ 1437us) tablet.cc:661] Decoding operations
0321 02:59:39.944379 (+ 15751us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:59:39.944471 (+ 92us) write_op.cc:641] Partition lock acquired for write op
0321 02:59:39.944505 (+ 34us) tablet.cc:684] Acquiring locks for 11 operations
0321 02:59:40.029481 (+ 84976us) tablet.cc:700] Row locks acquired
0321 02:59:40.029504 (+ 23us) write_op.cc:260] PREPARE: finished
0321 02:59:40.029674 (+ 170us) write_op.cc:270] Start()
0321 02:59:40.029833 (+ 159us) write_op.cc:276] Timestamp: P: 1774061980029612 usec, L: 0
0321 02:59:40.029855 (+ 22us) op_driver.cc:348] REPLICATION: starting
0321 02:59:40.030573 (+ 718us) log.cc:844] Serialized 4297 byte log entry
0321 02:59:40.240487 (+209914us) op_driver.cc:464] REPLICATION: finished
0321 02:59:40.463362 (+222875us) write_op.cc:301] APPLY: starting
0321 02:59:40.463524 (+ 162us) tablet.cc:1366] starting BulkCheckPresence
0321 02:59:40.463933 (+ 409us) tablet.cc:1369] finished BulkCheckPresence
0321 02:59:40.464118 (+ 185us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:59:40.465599 (+ 1481us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:59:40.465998 (+ 399us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0321 02:59:40.466053 (+ 55us) write_op.cc:312] APPLY: finished
0321 02:59:40.466445 (+ 392us) log.cc:844] Serialized 83 byte log entry
0321 02:59:40.466683 (+ 238us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:40.466829 (+ 146us) write_op.cc:454] Released schema lock
0321 02:59:40.466899 (+ 70us) write_op.cc:341] FINISH: Updating metrics
0321 02:59:40.570266 (+103367us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:40.570308 (+ 42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":42863,"reported_tablets":15,"spinlock_wait_cycles":1113934208,"child_traces":[["op",{"apply.queue_time_us":189519,"apply.run_cpu_time_us":4128,"apply.run_wall_time_us":115631,"num_ops":11,"prepare.queue_time_us":63878,"prepare.run_cpu_time_us":6444,"prepare.run_wall_time_us":213662,"replication_time_us":210427,"thread_start_us":165143,"threads_started":2}]]}
I20260321 02:59:40.859987 11200 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:40.850570 14209 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
W20260321 02:59:40.862313 11943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:40.853147 13723 raft_consensus.cc:2749] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:40.864895 11275 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:40.852267 14208 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:40.846626 14151 consensus_queue.cc:1048] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260321 02:59:40.868525 13933 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:40.869340 14074 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:40.845009 14162 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:40.847468 14200 consensus_queue.cc:1059] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.016s
I20260321 02:59:40.860538 13664 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:40.877205 13593 raft_consensus.cc:2804] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:40.869400 14215 consensus_queue.cc:237] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:40.888091 11130 catalog_manager.cc:5671] T c104d8b810534a6c918eab2f919a83dc P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 4 to 6, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 6 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:40.891503 13664 raft_consensus.cc:3060] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:40.898190 11130 catalog_manager.cc:5671] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:40.904683 11130 catalog_manager.cc:5671] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:59:40.875165 14225 raft_consensus.cc:2804] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:40.910677 14225 raft_consensus.cc:697] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:40.910871 11302 raft_consensus.cc:1275] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 8 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260321 02:59:40.911722 14225 consensus_queue.cc:237] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:40.882844 14024 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:40.913609 11302 raft_consensus.cc:1275] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260321 02:59:40.890683 14208 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:40.893771 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:40.914834 11973 raft_consensus.cc:3060] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:40.915081 11130 catalog_manager.cc:5671] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:59:40.846800 14163 consensus_queue.cc:1048] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20260321 02:59:40.905735 14162 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:40.908288 14151 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:40.920617 14151 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:40.921165 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:40.921774 11302 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:40.922770 14151 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:40.908646 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:40.881821 13865 raft_consensus.cc:2749] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:40.923790 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:40.909782 11974 raft_consensus.cc:2468] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 7.
I20260321 02:59:40.883628 11275 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:40.926385 13865 raft_consensus.cc:2749] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260321 02:59:40.916160 13593 raft_consensus.cc:697] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:40.916024 14208 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:40.894042 14200 raft_consensus.cc:3055] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:40.927493 14200 raft_consensus.cc:740] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:40.927722 13593 consensus_queue.cc:237] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:40.928437 14200 consensus_queue.cc:260] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:40.929749 14200 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Advancing to term 7
W20260321 02:59:40.917169 11942 leader_election.cc:336] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.709s (SENT)
I20260321 02:59:40.917997 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:40.930768 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 47223.2us
I20260321 02:59:40.938180 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:40.941313 11380 raft_consensus.cc:1240] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:40.942009 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:40.942620 11383 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:40.942889 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:40.943696 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:40.943531 11302 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:40.937354 14163 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:40.947788 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:40.948220 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:40.955935 14076 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:40.956683 11973 raft_consensus.cc:2468] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:40.956881 14076 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:40.936297 13865 consensus_queue.cc:1048] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260321 02:59:40.957653 11350 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:40.959162 14076 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:40.952023 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:40.950959 11130 catalog_manager.cc:5671] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 4 to 6, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 6 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:40.970162 11381 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:40.964754 11310 raft_consensus.cc:1240] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:40.968924 11308 raft_consensus.cc:2393] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 7: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:40.954113 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:40.973551 11276 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:40.954010 14215 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:40.974726 14208 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:40.975368 14208 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:40.976915 14209 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:40.978272 14208 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:40.979506 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:40.980351 11311 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:40.951325 13664 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:59:40.982750 13865 raft_consensus.cc:2804] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:40.982997 14024 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:40.983381 13865 raft_consensus.cc:697] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:40.983512 14024 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:40.991305 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:40.975200 14215 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:40.993948 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:40.996981 13664 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:40.979780 14209 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:40.997982 14215 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:40.984937 13865 consensus_queue.cc:237] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:40.959906 14076 raft_consensus.cc:2804] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:41.003317 14024 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:41.007436 11311 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:40.940668 14027 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:41.009270 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:41.003973 14076 raft_consensus.cc:697] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [term 7 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:41.011765 11976 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:41.014815 11276 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:40.948341 14163 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:41.026652 11130 catalog_manager.cc:5671] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 6 to 7, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 7 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:41.031608 14226 consensus_queue.cc:1048] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260321 02:59:41.036294 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:41.037351 11304 raft_consensus.cc:2393] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 7: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:41.038617 14027 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 10 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:41.040479 14163 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:41.042160 11377 raft_consensus.cc:1240] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:41.043205 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:41.027706 14215 consensus_queue.cc:1048] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260321 02:59:41.043923 11377 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
W20260321 02:59:41.044083 11943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:41.044377 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:41.052747 11130 catalog_manager.cc:5671] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 5 to 7, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 7 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:41.057868 11976 raft_consensus.cc:1275] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260321 02:59:41.060020 11130 catalog_manager.cc:5671] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:41.084527 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:41.048831 13933 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:41.100385 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 84165.5us
I20260321 02:59:41.092900 11130 catalog_manager.cc:5671] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 4 to 8, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 8 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:41.107378 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:41.111400 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:41.109130 11377 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
W20260321 02:59:41.111930 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 106516us
I20260321 02:59:41.116154 11130 catalog_manager.cc:5671] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:41.092320 14162 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:41.124765 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:41.046036 14076 consensus_queue.cc:237] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:41.126304 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:41.131362 11130 catalog_manager.cc:5671] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: config changed from index -1 to 5. New cstate: current_term: 7 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } health_report { overall_health: HEALTHY } } }
I20260321 02:59:41.115496 14091 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:41.092077 14207 consensus_queue.cc:1048] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.037s
W20260321 02:59:41.142402 11940 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:41.102792 13933 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:41.151446 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:41.154937 11311 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 8.
I20260321 02:59:41.131812 13593 consensus_queue.cc:1059] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260321 02:59:41.131570 14208 consensus_queue.cc:1048] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260321 02:59:41.132941 14162 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:41.207881 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:41.210144 11130 catalog_manager.cc:5671] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:41.150858 14209 consensus_queue.cc:1059] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.026s
I20260321 02:59:41.150599 14091 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:41.217126 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:41.218144 11307 raft_consensus.cc:2393] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
W20260321 02:59:41.226874 14238 log.cc:927] Time spent T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
W20260321 02:59:41.228260 14233 log.cc:927] Time spent T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
W20260321 02:59:41.234256 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 68515.9us
I20260321 02:59:41.237363 11382 raft_consensus.cc:1275] T 7d5b2547bde048c88e963f650d905f76 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260321 02:59:41.246079 11130 catalog_manager.cc:5671] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:59:41.253244 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:41.254091 11311 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 8.
I20260321 02:59:41.269483 11308 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 7.
I20260321 02:59:41.258369 11130 catalog_manager.cc:5671] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:41.280073 11130 catalog_manager.cc:5671] T 3c50eb057ef548839834ecb7dfd20ce5 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:41.285230 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:41.274876 14209 raft_consensus.cc:993] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb: : Instructing follower 83b12b883a7f4129a87dadf5a78f08da to start an election
I20260321 02:59:41.276072 14024 raft_consensus.cc:2804] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:41.251096 14119 consensus_queue.cc:1048] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260321 02:59:41.278530 14208 raft_consensus.cc:3055] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Stepping down as leader of term 9
I20260321 02:59:41.294504 11130 catalog_manager.cc:5671] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 5 to 7, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 7 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:41.299110 11130 catalog_manager.cc:5671] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb reported cstate change: term changed from 6 to 8, leader changed from fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 8 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:41.313441 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:41.316555 14241 log.cc:927] Time spent T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260321 02:59:41.319386 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:41.317957 13933 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:41.317879 14163 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:41.299798 14208 raft_consensus.cc:740] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
W20260321 02:59:41.304845 14232 log.cc:927] Time spent T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.173s user 0.001s sys 0.000s
I20260321 02:59:41.294412 14250 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:41.356276 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:41.356449 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 52789.1us
W20260321 02:59:41.362152 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 82794.2us
I20260321 02:59:41.398702 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:41.359082 14162 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:41.354317 13723 raft_consensus.cc:1081] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 LEADER]: Signalling peer 83b12b883a7f4129a87dadf5a78f08da to start an election
I20260321 02:59:41.355649 14151 consensus_queue.cc:1048] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
W20260321 02:59:41.425791 11943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:41.432547 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:41.355969 14163 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:41.437513 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:41.355319 14250 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:41.440418 14245 log.cc:927] Time spent T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
W20260321 02:59:41.440784 14235 log.cc:927] Time spent T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.193s user 0.001s sys 0.000s
W20260321 02:59:41.485399 11943 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:41.441090 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:41.441486 14261 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:41.441231 14263 consensus_queue.cc:1048] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260321 02:59:41.444728 14163 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:41.433377 14091 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:41.449699 11302 raft_consensus.cc:1275] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260321 02:59:41.495419 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:41.496698 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:41.497491 11302 raft_consensus.cc:2393] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 6: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:41.496905 14261 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:41.498274 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:41.499073 11302 raft_consensus.cc:2393] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
I20260321 02:59:41.450738 11310 raft_consensus.cc:1275] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260321 02:59:41.428269 14208 consensus_queue.cc:260] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:41.451346 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:41.501012 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:41.452076 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:41.435703 14234 log.cc:927] Time spent T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
I20260321 02:59:41.453500 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:41.502440 11232 raft_consensus.cc:1275] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260321 02:59:41.502810 11232 pending_rounds.cc:85] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb: Aborting all ops after (but not including) 3
I20260321 02:59:41.502864 11306 raft_consensus.cc:2393] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 10: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:41.454695 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:41.454926 14236 log.cc:927] Time spent T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.287s user 0.003s sys 0.000s
I20260321 02:59:41.463532 11308 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 8.
I20260321 02:59:41.480779 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:41.482672 11200 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:41.493348 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:41.503118 11232 pending_rounds.cc:107] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260321 02:59:41.509608 11232 raft_consensus.cc:2889] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:41.508342 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:41.501015 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:41.499780 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:41.511355 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:41.515128 11200 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:41.515661 11976 raft_consensus.cc:2393] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
I20260321 02:59:41.512123 11307 raft_consensus.cc:2376] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6. Re-sending same reply.
I20260321 02:59:41.520872 11977 raft_consensus.cc:1275] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260321 02:59:41.525398 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 62596.6us
I20260321 02:59:41.528273 11232 raft_consensus.cc:1275] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260321 02:59:41.499861 14261 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:41.530207 11349 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:41.523013 13593 raft_consensus.cc:2749] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:41.532951 11233 raft_consensus.cc:1275] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260321 02:59:41.537942 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:41.538594 11233 raft_consensus.cc:2468] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:41.526786 14027 raft_consensus.cc:2804] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:41.542029 14091 raft_consensus.cc:2749] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260321 02:59:41.560635 11275 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:41.562299 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:41.563190 11233 raft_consensus.cc:2393] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 6: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:41.554906 14163 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:41.566238 14163 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:41.566776 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:41.566908 14250 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:41.564724 11227 raft_consensus.cc:1275] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260321 02:59:41.570765 11942 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:41.574955 13933 consensus_queue.cc:1048] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:59:41.576105 14024 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:41.576601 14024 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:41.578112 14200 raft_consensus.cc:2749] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260321 02:59:41.546061 14027 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:41.579677 14027 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:41.581825 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:41.584124 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:41.585645 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:41.586439 11232 raft_consensus.cc:3060] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 7
I20260321 02:59:41.560542 14162 consensus_queue.cc:1048] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260321 02:59:41.575119 14122 raft_consensus.cc:2749] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260321 02:59:41.609099 14163 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:41.610409 14118 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:41.611146 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:41.611987 11236 raft_consensus.cc:2393] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 8: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:41.612180 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:41.612741 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:41.613549 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:41.614187 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:41.614558 11230 raft_consensus.cc:2393] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 6: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:41.616070 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:41.617967 14272 log.cc:927] Time spent T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260321 02:59:41.611994 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:41.611012 14118 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:41.627661 11233 raft_consensus.cc:1275] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:41.613598 11229 raft_consensus.cc:2410] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260321 02:59:41.588302 11349 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:41.631433 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:41.634014 14276 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:41.638226 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:41.634102 14122 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:41.635859 14276 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:41.641160 14122 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:41.643110 14122 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:41.630602 13846 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:41.647540 11232 raft_consensus.cc:2468] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:41.652314 11236 raft_consensus.cc:1240] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 5. Current term is 6. Ops: []
W20260321 02:59:41.657127 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 96589.1us
I20260321 02:59:41.656724 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:41.658108 11275 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:41.660148 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 98807.8us
I20260321 02:59:41.664565 14274 raft_consensus.cc:2749] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:41.665577 14118 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:41.643133 14276 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:41.667901 11349 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:41.655289 14091 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:41.653251 14122 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:41.669389 11275 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:41.671872 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:41.636062 14027 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:41.674248 11349 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:41.676604 14251 log.cc:927] Time spent T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.095s user 0.001s sys 0.004s
I20260321 02:59:41.678712 11377 raft_consensus.cc:1275] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260321 02:59:41.680243 14255 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:41.680872 14255 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:41.681216 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:41.682806 14255 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:41.682996 11381 raft_consensus.cc:1275] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260321 02:59:41.683687 14118 raft_consensus.cc:2804] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:41.643911 13846 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:41.689990 11275 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:41.671934 13865 raft_consensus.cc:2749] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260321 02:59:41.699995 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:41.700394 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:41.700897 13933 raft_consensus.cc:2749] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260321 02:59:41.697394 14276 raft_consensus.cc:2749] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:41.701432 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:41.702206 11275 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:41.688163 14118 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:41.702980 14276 raft_consensus.cc:2749] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260321 02:59:41.703212 14118 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:41.704656 13865 consensus_queue.cc:1048] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:59:41.688491 14274 raft_consensus.cc:2804] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:41.707006 14274 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:41.707487 14274 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:41.711330 11381 raft_consensus.cc:1275] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260321 02:59:41.712605 11275 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:41.713762 11232 raft_consensus.cc:1240] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:41.715054 11377 raft_consensus.cc:1275] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260321 02:59:41.715791 14276 consensus_queue.cc:1048] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.002s
I20260321 02:59:41.716146 13933 raft_consensus.cc:2749] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:41.716825 13933 consensus_queue.cc:1048] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:59:41.722611 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:41.723420 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:41.724069 11229 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 9.
I20260321 02:59:41.724116 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:41.725112 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:41.725718 11233 raft_consensus.cc:2468] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:41.727465 13846 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:41.727519 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:41.698637 14208 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:41.741835 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:41.728080 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:41.674104 14278 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:41.729378 11236 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 8.
I20260321 02:59:41.746552 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:41.749032 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:41.727634 14027 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:41.752235 11235 raft_consensus.cc:1275] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260321 02:59:41.753366 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:41.755640 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:41.756040 14278 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:41.757408 11385 raft_consensus.cc:1275] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:41.757860 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:41.758761 14278 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:41.758992 11228 raft_consensus.cc:1240] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:41.760004 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:41.757447 11383 raft_consensus.cc:1275] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260321 02:59:41.760778 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:41.762346 11377 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:41.760748 11376 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:41.764542 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:41.689043 14122 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:41.768124 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:41.765314 11234 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 7.
I20260321 02:59:41.768945 11302 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:41.684816 14091 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:41.793982 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:41.794377 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:41.796733 11377 raft_consensus.cc:3060] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:41.796901 11303 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:41.795509 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:41.794961 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:41.803329 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:41.804241 11383 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:41.808583 11200 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:41.810988 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 95847.2us
I20260321 02:59:41.817154 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:41.817727 11383 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:41.789134 14027 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:41.788724 14278 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:41.824115 11376 raft_consensus.cc:1275] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260321 02:59:41.823796 14266 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:41.827768 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:41.828490 11382 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:41.828347 11200 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:41.827013 14266 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:41.838490 14209 raft_consensus.cc:2804] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:41.841130 11385 raft_consensus.cc:1275] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 8 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260321 02:59:41.845690 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:41.820667 14257 raft_consensus.cc:2804] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:41.815476 13846 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:41.852022 11380 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:41.846143 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:42.055758 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 214671us
I20260321 02:59:42.058144 11977 raft_consensus.cc:1275] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260321 02:59:42.058410 11303 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 10.
I20260321 02:59:41.952459 14118 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:42.060551 14118 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:42.056691 11200 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:42.013756 14207 raft_consensus.cc:3055] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:42.061596 14207 raft_consensus.cc:740] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:42.061851 14271 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260321 02:59:42.062337 14271 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:41.852741 14074 consensus_queue.cc:1059] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.210s
I20260321 02:59:42.062675 11200 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:41.891846 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:41.926901 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:42.066654 11385 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:42.056083 11376 raft_consensus.cc:2468] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 5.
I20260321 02:59:42.071473 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:42.043828 14279 log.cc:927] Time spent T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.184s user 0.001s sys 0.000s
W20260321 02:59:41.920307 11349 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.9.213.1:36717 timed out after 1.821s (SENT) [suppressed 34 similar messages]
I20260321 02:59:41.839828 14098 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:42.072157 11306 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
W20260321 02:59:42.029065 14280 log.cc:927] Time spent T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c: Append to log took a long time: real 0.374s user 0.000s sys 0.003s
I20260321 02:59:42.043038 13664 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:42.056584 14257 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:41.833036 14255 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:41.845685 14209 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:42.062822 14271 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Advancing to term 10
W20260321 02:59:42.078881 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 158463us
I20260321 02:59:41.829555 14278 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:42.085650 13664 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:42.088254 14262 raft_consensus.cc:2804] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:41.860675 14027 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:41.839515 14215 consensus_queue.cc:1059] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.246s
I20260321 02:59:41.919735 14276 consensus_queue.cc:1048] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.172s
I20260321 02:59:42.107986 11349 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:42.097477 14257 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:42.044346 14282 consensus_queue.cc:1048] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.190s
I20260321 02:59:42.083858 14098 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:42.071458 14118 consensus_queue.cc:1048] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260321 02:59:42.105746 14209 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:42.112824 14262 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:42.142040 11349 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:42.079234 14024 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:42.101128 14207 consensus_queue.cc:260] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:42.145345 14027 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:42.108259 14122 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:42.142478 14274 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:42.149979 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 334702us
I20260321 02:59:42.074893 13846 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:42.174131 11378 raft_consensus.cc:1240] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260321 02:59:42.103396 14255 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:42.168210 14098 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:42.172982 11306 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba"
dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:56630
I20260321 02:59:42.181358 11275 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:42.132609 14091 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:42.149433 14266 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:42.192651 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 50672.1us
I20260321 02:59:42.171732 14262 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:42.222201 11349 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:42.237455 11305 raft_consensus.cc:1275] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260321 02:59:42.237385 11349 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:42.238094 11305 pending_rounds.cc:85] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da: Aborting all ops after (but not including) 3
I20260321 02:59:42.247136 11305 pending_rounds.cc:107] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260321 02:59:42.247495 11305 raft_consensus.cc:2889] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:42.249351 11349 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:42.253893 11349 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:42.264629 11232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
W20260321 02:59:42.266062 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 136189us
I20260321 02:59:42.272308 11306 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:59:42.273080 11306 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Advancing to term 11
I20260321 02:59:42.240065 14091 raft_consensus.cc:2749] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260321 02:59:42.265347 11232 raft_consensus.cc:606] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:42.220724 14289 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:42.288154 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:42.287432 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 159732us
I20260321 02:59:42.289425 11303 raft_consensus.cc:2468] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:42.256479 14281 raft_consensus.cc:2749] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:42.260828 14276 raft_consensus.cc:2749] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:42.256659 14076 raft_consensus.cc:2749] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260321 02:59:42.264436 14207 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:42.272784 14163 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260321 02:59:42.299244 11200 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:42.321421 11200 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:42.326363 11200 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
W20260321 02:59:42.336642 11377 consensus_meta.cc:287] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.539s user 0.010s sys 0.001s
I20260321 02:59:42.337141 11377 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:42.343590 11200 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:42.310639 14163 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:42.374615 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:42.286303 14295 consensus_queue.cc:1048] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.086s
W20260321 02:59:42.385313 14286 log.cc:927] Time spent T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.136s user 0.000s sys 0.001s
I20260321 02:59:42.338524 14291 raft_consensus.cc:2749] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:42.359493 14163 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:42.296147 14293 consensus_queue.cc:1059] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.083s
I20260321 02:59:42.381817 14266 raft_consensus.cc:2749] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260321 02:59:42.427273 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 83806.7us
I20260321 02:59:42.381678 13664 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:42.434576 11200 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:42.455687 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:42.456398 11974 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:42.425629 14074 raft_consensus.cc:2749] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:42.424075 14281 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:42.425172 14098 raft_consensus.cc:2804] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:42.455253 14302 raft_consensus.cc:2804] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:42.427594 14024 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:42.460541 14090 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:42.461620 14287 log.cc:927] Time spent T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.185s user 0.000s sys 0.001s
W20260321 02:59:42.461685 14301 log.cc:927] Time spent T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20260321 02:59:42.462558 14303 log.cc:927] Time spent T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20260321 02:59:42.427379 14122 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:42.444463 14293 consensus_queue.cc:1243] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer 0d7efc590a1041a9bd4cd4a971332dfb log is divergent from this leader: its last log entry 4.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20260321 02:59:42.492049 11382 consensus_meta.cc:287] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.638s user 0.007s sys 0.006s
I20260321 02:59:42.438248 14278 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:42.437327 14276 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:42.464076 13846 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:42.504915 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 70475.3us
I20260321 02:59:42.504663 11382 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 10.
I20260321 02:59:42.486497 14098 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:42.485775 14302 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:42.493451 14274 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:42.539564 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:42.535784 14098 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:42.546856 14107 raft_consensus.cc:2804] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:42.590245 14107 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:42.590896 14107 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:42.555967 14255 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:42.499747 14090 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:42.597220 11232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2d71c397418a46d085c9974230b7f964"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:42.564999 14227 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:42.597918 11232 raft_consensus.cc:606] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:42.598270 14227 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:42.597218 14090 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:42.600601 14227 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:42.602383 14309 raft_consensus.cc:3055] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:42.602874 14309 raft_consensus.cc:740] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:42.609119 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:42.609841 11378 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 7.
I20260321 02:59:42.610265 11974 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:42.568971 14274 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:42.612222 11276 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:42.533052 14276 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:42.510187 14122 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:42.491161 14281 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:42.577003 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 53067us
I20260321 02:59:42.578850 14226 raft_consensus.cc:3055] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:42.538760 14302 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:42.584543 14162 raft_consensus.cc:3055] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:42.615854 11942 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:42.607349 11306 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:42.628491 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:42.629758 11232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:42.630667 11232 raft_consensus.cc:606] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:42.629508 14162 raft_consensus.cc:740] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:42.654850 14317 raft_consensus.cc:2804] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:42.627784 14226 raft_consensus.cc:740] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:42.661151 11379 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 7.
W20260321 02:59:42.661620 11349 leader_election.cc:336] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.794s (SENT)
I20260321 02:59:42.621198 14282 raft_consensus.cc:2804] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election won for term 8
W20260321 02:59:42.674221 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 482678us
I20260321 02:59:42.682088 11349 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:42.664700 11942 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:42.675107 11200 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:42.686496 11200 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:42.687497 11200 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:42.686717 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:42.622165 14311 consensus_queue.cc:1048] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
I20260321 02:59:42.696429 11200 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:42.701493 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:42.702502 11349 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:42.682590 13933 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:42.629139 14274 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:42.664426 14317 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:42.707188 11349 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:42.702348 11307 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 6.
I20260321 02:59:42.706377 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:42.695889 14090 raft_consensus.cc:2804] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:42.705448 11200 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:42.691936 14215 raft_consensus.cc:2804] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:42.691359 14282 raft_consensus.cc:697] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
W20260321 02:59:42.709131 14208 consensus_meta.cc:287] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.958s user 0.011s sys 0.002s
I20260321 02:59:42.697793 14074 raft_consensus.cc:2804] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:42.721490 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:42.682909 14209 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:42.689319 14319 raft_consensus.cc:2804] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:42.724252 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:42.634321 14309 consensus_queue.cc:260] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:42.714150 14317 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:42.708802 14076 raft_consensus.cc:2749] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260321 02:59:42.773872 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 48121.1us
I20260321 02:59:42.708527 14255 raft_consensus.cc:2749] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260321 02:59:42.781921 11230 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 10.
I20260321 02:59:42.711068 14119 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:42.708612 14091 consensus_queue.cc:1048] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.066s
I20260321 02:59:42.795902 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:42.780277 13723 consensus_queue.cc:1059] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260321 02:59:42.797132 13723 raft_consensus.cc:3055] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Stepping down as leader of term 8
I20260321 02:59:42.797504 13723 raft_consensus.cc:740] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:42.797955 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:42.777657 14309 raft_consensus.cc:3060] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:42.798276 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:42.798758 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:42.784089 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:42.798947 11308 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:42.799408 11383 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:42.800590 11304 raft_consensus.cc:1275] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260321 02:59:42.800827 11383 raft_consensus.cc:1240] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:42.705194 14162 consensus_queue.cc:260] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:42.715987 14118 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:42.724983 11235 raft_consensus.cc:1275] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260321 02:59:42.715281 14090 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:42.796391 14255 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:42.798054 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:42.872653 11307 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:42.773171 14278 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:42.772295 14304 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Leader election won for term 10
W20260321 02:59:42.875149 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 86228.7us
I20260321 02:59:42.798907 11310 raft_consensus.cc:2393] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 6: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:42.873643 14278 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:42.799435 11378 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:42.771550 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:42.784910 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:42.779135 11232 raft_consensus.cc:1275] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260321 02:59:42.878751 11382 raft_consensus.cc:2393] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 6: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:42.662825 14315 consensus_queue.cc:1048] T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.075s
W20260321 02:59:42.879861 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 75888.4us
I20260321 02:59:42.880789 11275 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:42.887795 14278 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:42.720644 14215 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:42.779101 14283 log.cc:927] Time spent T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.772s user 0.000s sys 0.004s
I20260321 02:59:42.806152 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:42.893587 14215 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:42.830742 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:42.894928 11307 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
I20260321 02:59:42.806200 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:42.786429 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:42.708855 14266 raft_consensus.cc:2804] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:42.902407 14215 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:42.904640 14215 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:42.915511 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:42.778460 14319 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:42.916482 11381 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:42.918856 11200 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:42.716336 13933 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:42.777273 14074 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:42.942030 14162 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:42.942571 11230 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58200 (request call id 1725) took 1220 ms. Trace:
I20260321 02:59:42.942987 11230 rpcz_store.cc:276] 0321 02:59:41.722394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:41.727815 (+ 5421us) service_pool.cc:225] Handling call
0321 02:59:42.942379 (+1214564us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540037a854. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:42.942489 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2388896000}
I20260321 02:59:42.912827 14090 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:42.944069 13846 consensus_queue.cc:1048] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
W20260321 02:59:42.944417 14207 consensus_meta.cc:287] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.627s user 0.008s sys 0.002s
W20260321 02:59:42.945349 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 64669.5us
W20260321 02:59:42.938366 14271 consensus_meta.cc:287] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.823s user 0.009s sys 0.003s
I20260321 02:59:42.954666 11309 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 7.
W20260321 02:59:42.955790 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 248654us
I20260321 02:59:42.960134 11275 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:42.961555 11275 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:42.962477 14293 consensus_queue.cc:1048] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:59:42.963881 14293 raft_consensus.cc:2749] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:42.964520 14293 raft_consensus.cc:2749] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:42.964509 14291 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:42.964898 11349 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:42.965227 14291 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:42.966478 14278 consensus_queue.cc:1048] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:59:42.966842 14076 raft_consensus.cc:2804] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:42.967309 14076 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:42.967731 14076 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:42.968693 14291 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:42.969100 11275 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:42.969826 14295 consensus_queue.cc:1048] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:59:42.970409 14278 raft_consensus.cc:2749] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:42.923599 14319 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:42.937852 14266 raft_consensus.cc:697] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:42.975651 11275 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:42.924350 14119 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:42.954408 14314 log.cc:927] Time spent T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.121s user 0.000s sys 0.000s
I20260321 02:59:42.950366 14215 raft_consensus.cc:2749] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:42.945315 14323 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:42.946978 14307 raft_consensus.cc:2749] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260321 02:59:42.946295 14074 raft_consensus.cc:3060] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:42.922024 14118 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:42.798282 13723 consensus_queue.cc:260] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:42.974884 14289 consensus_meta.cc:287] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.681s user 0.011s sys 0.000s
I20260321 02:59:42.954813 14304 raft_consensus.cc:697] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:43.009351 14266 consensus_queue.cc:237] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:59:43.010195 14304 consensus_queue.cc:237] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:42.985414 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:42.981942 14298 consensus_queue.cc:1048] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260321 02:59:42.986609 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:43.012823 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:43.013473 11382 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:43.014003 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:43.014344 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:43.014673 11307 raft_consensus.cc:2376] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 8. Re-sending same reply.
I20260321 02:59:43.015686 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:43.016556 11307 raft_consensus.cc:2393] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 10: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:42.973701 14271 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.017382 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:43.018124 14118 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:43.018536 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:42.986627 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:43.018786 14271 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 10 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:43.019934 11234 raft_consensus.cc:1240] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 9. Current term is 10. Ops: []
I20260321 02:59:42.986001 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:42.987381 11384 raft_consensus.cc:1275] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:42.988184 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:43.021924 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:42.722401 14226 consensus_queue.cc:260] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:42.989447 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:42.990279 14257 consensus_meta.cc:287] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.854s user 0.008s sys 0.004s
I20260321 02:59:42.990864 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:42.990989 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:42.996508 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:42.999244 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:42.990403 11231 raft_consensus.cc:1240] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260321 02:59:43.003286 11349 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:42.956046 14282 consensus_queue.cc:237] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:43.008605 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:43.012253 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:42.914867 14302 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.023039 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:42.958860 14276 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:42.988780 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:43.025338 11383 raft_consensus.cc:1275] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260321 02:59:43.035903 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:43.036816 14255 raft_consensus.cc:2749] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:43.037686 14255 raft_consensus.cc:2749] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260321 02:59:43.038168 11378 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:43.040217 14331 raft_consensus.cc:2749] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:43.042484 14319 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.001452 14122 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:43.007637 14323 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:43.044591 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:43.045010 11310 raft_consensus.cc:3060] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:43.045784 11231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:43.046139 14323 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:43.046327 11231 raft_consensus.cc:606] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:43.006428 14291 consensus_queue.cc:1059] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260321 02:59:43.047346 14327 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:43.047431 14255 raft_consensus.cc:3055] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 LEADER]: Stepping down as leader of term 8
I20260321 02:59:43.027740 14289 raft_consensus.cc:2749] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260321 02:59:43.027535 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 123890us
I20260321 02:59:43.049209 11377 raft_consensus.cc:1240] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:43.049966 14090 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:43.027316 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:43.048272 14255 raft_consensus.cc:740] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:43.053383 11310 raft_consensus.cc:1275] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260321 02:59:43.053718 11378 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:43.033896 11305 raft_consensus.cc:2393] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 8: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
W20260321 02:59:43.026103 14163 consensus_meta.cc:287] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.602s user 0.009s sys 0.002s
I20260321 02:59:42.980525 14295 consensus_queue.cc:1048] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.049s
I20260321 02:59:43.094022 11974 raft_consensus.cc:1275] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260321 02:59:43.094671 14327 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.095180 14337 consensus_queue.cc:1059] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20260321 02:59:43.095913 14281 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:43.097054 14300 log.cc:927] Time spent T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.610s user 0.004s sys 0.000s
I20260321 02:59:43.098238 14319 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:43.100989 11228 raft_consensus.cc:1240] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260321 02:59:43.101356 14302 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:43.021991 14274 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:43.105945 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 102777us
I20260321 02:59:43.109580 11231 raft_consensus.cc:1240] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:43.136826 11349 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
W20260321 02:59:43.132658 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 90457.5us
W20260321 02:59:43.141052 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 53226us
W20260321 02:59:43.142560 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 109953us
I20260321 02:59:43.145457 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:43.104943 14226 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:43.056517 14118 consensus_queue.cc:1048] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.089s
I20260321 02:59:43.163206 13846 consensus_queue.cc:1059] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:59:43.163560 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:43.163690 11381 raft_consensus.cc:1275] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260321 02:59:43.164366 11383 raft_consensus.cc:2393] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 7: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:43.165696 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:43.166723 11378 raft_consensus.cc:2393] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 7: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:43.169467 11376 raft_consensus.cc:1275] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260321 02:59:43.170033 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:43.170719 11380 raft_consensus.cc:2468] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:43.173831 13846 raft_consensus.cc:3055] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:43.174256 13846 raft_consensus.cc:740] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:43.178350 14296 consensus_queue.cc:1059] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:43.175355 13846 consensus_queue.cc:260] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.180028 13846 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:43.188283 14295 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:43.188935 14295 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.191501 14295 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:43.192907 11275 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:43.192983 14296 raft_consensus.cc:3055] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:43.193495 14296 raft_consensus.cc:740] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:43.194340 14296 consensus_queue.cc:260] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.196058 14296 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:43.197568 11132 catalog_manager.cc:5671] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 7 to 8, leader changed from 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 8 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:59:43.200500 14295 raft_consensus.cc:2749] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:43.200155 11132 catalog_manager.cc:5671] T 7d5b2547bde048c88e963f650d905f76 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:59:43.202279 11275 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:43.203119 14295 raft_consensus.cc:2804] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:43.203557 14295 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:43.203967 14295 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:43.216840 11231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:43.217820 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:43.218983 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 60205.5us
I20260321 02:59:43.034660 14257 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.217639 11231 raft_consensus.cc:606] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Received request to transfer leadership to TS 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:43.225958 11275 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:43.227062 13865 raft_consensus.cc:2749] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:43.153272 14209 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:43.234104 11275 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:43.232776 14024 consensus_queue.cc:1048] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.004s
I20260321 02:59:43.143580 14122 consensus_queue.cc:1048] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.077s
I20260321 02:59:43.240619 14331 consensus_queue.cc:1048] T 00eeadd5c566459b86e97f920a91c398 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260321 02:59:43.240783 14076 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.242597 14119 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:43.162161 14274 raft_consensus.cc:2804] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:43.244599 14299 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:43.244886 14274 raft_consensus.cc:697] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 8 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:43.245263 14299 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:43.245844 14274 consensus_queue.cc:237] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.247455 14299 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:43.247860 11378 raft_consensus.cc:1240] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260321 02:59:43.248471 14090 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:43.251477 14325 log.cc:927] Time spent T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.473s user 0.012s sys 0.016s
I20260321 02:59:43.254220 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260321 02:59:43.254628 11306 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 11 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:43.255189 11306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:56630 (request call id 1861) took 1149 ms. Trace:
I20260321 02:59:43.256419 14255 consensus_queue.cc:260] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:43.102648 14163 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.231029 11349 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:43.228087 14262 consensus_meta.cc:287] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.967s user 0.006s sys 0.006s
I20260321 02:59:43.319156 11230 raft_consensus.cc:1240] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9. Ops: [8.5-8.5]
W20260321 02:59:43.300330 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 66310.3us
W20260321 02:59:43.302050 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 71108.8us
I20260321 02:59:43.320555 11349 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:43.321918 14289 raft_consensus.cc:2749] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:43.255523 11306 rpcz_store.cc:276] 0321 02:59:42.121294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:42.127445 (+ 6151us) service_pool.cc:225] Handling call
0321 02:59:43.253454 (+1126009us) spinlock_profiling.cc:243] Waited 89.7 ms on lock 0x7b54002102d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac543e1d 00007fa8b1510390 00007fa8aaa9b3b4 00007fa8aaa9b342 00007fa8aaa9b271 00007fa8aaa9b1ec 00007fa8aaa9a4a2 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:43.255092 (+ 1638us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":206281600}
I20260321 02:59:43.306447 14295 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.258178 14076 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:43.315372 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 225980us
I20260321 02:59:43.152942 14309 consensus_queue.cc:1048] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.080s
W20260321 02:59:43.328902 11942 leader_election.cc:336] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267): Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.777s (SENT)
I20260321 02:59:43.328186 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:43.332718 11378 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:43.333834 14255 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:43.335534 14343 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:43.336316 14340 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:43.336912 14340 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:43.339015 14340 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:43.340394 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:43.342007 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 118338us
I20260321 02:59:43.342727 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:59:43.320598 11275 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:43.343502 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:43.356585 11942 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:43.361102 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:43.370066 11378 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 8.
W20260321 02:59:43.371608 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 51066.2us
I20260321 02:59:43.372442 11275 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:43.324739 14098 consensus_meta.cc:287] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.735s user 0.008s sys 0.003s
W20260321 02:59:43.330534 14317 consensus_meta.cc:287] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 0.547s user 0.008s sys 0.003s
I20260321 02:59:43.319021 14307 raft_consensus.cc:2749] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260321 02:59:43.373867 11275 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:43.336167 14343 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.325193 14091 raft_consensus.cc:2749] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260321 02:59:43.385747 11275 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:43.228261 14249 consensus_queue.cc:1048] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.142s
I20260321 02:59:43.399982 11275 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:43.404815 11130 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42502 (request call id 226) took 7428 ms. Trace:
W20260321 02:59:43.404537 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 75641.3us
W20260321 02:59:43.400121 14280 log.cc:927] Time spent T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
I20260321 02:59:43.372680 13933 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:43.375856 13846 raft_consensus.cc:2749] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:43.376129 14331 raft_consensus.cc:2749] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:43.376183 14299 raft_consensus.cc:2749] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260321 02:59:43.331306 14262 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.420557 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:43.421612 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:43.380353 14098 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.421772 14163 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:43.385360 14317 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:43.426748 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:43.453276 11305 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 7.
I20260321 02:59:43.427146 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:43.409017 14307 raft_consensus.cc:2804] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260321 02:59:43.438832 11311 raft_consensus.cc:2410] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260321 02:59:43.395920 14278 raft_consensus.cc:2749] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:43.430884 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:43.440531 11942 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:43.379694 14296 consensus_queue.cc:1048] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260321 02:59:43.455525 11306 raft_consensus.cc:2468] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 5.
I20260321 02:59:43.381220 14074 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.463423 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:43.472754 11378 raft_consensus.cc:2468] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:43.418756 14024 consensus_queue.cc:1059] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260321 02:59:43.473840 14296 raft_consensus.cc:3055] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Stepping down as leader of term 8
I20260321 02:59:43.473505 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:43.474304 11201 consensus_peers.cc:597] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb -> Peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285): Couldn't send request to peer 3d8d3176d6ec407ea2a965556c2e2c56. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f4ec8d11b7b4e6096d8c2b8ece078dc. This is attempt 1: this message will repeat every 5th retry.
I20260321 02:59:43.474414 14296 raft_consensus.cc:740] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:43.421635 11973 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45088 (request call id 314) took 1146 ms. Trace:
I20260321 02:59:43.475323 14296 consensus_queue.cc:260] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.476581 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:43.477250 11235 raft_consensus.cc:2468] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:43.416826 14122 raft_consensus.cc:2749] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:43.399101 14282 consensus_queue.cc:1048] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260321 02:59:43.473521 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: true dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:43.483920 11228 raft_consensus.cc:3055] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 LEADER]: Stepping down as leader of term 10
I20260321 02:59:43.484297 11228 raft_consensus.cc:740] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:43.484539 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:43.485807 11977 raft_consensus.cc:2393] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 8: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
I20260321 02:59:43.407655 11130 rpcz_store.cc:276] 0321 02:59:36.013483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:36.082323 (+ 68840us) service_pool.cc:225] Handling call
0321 02:59:37.788148 (+1705825us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54002d69d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b6470 00007fa8ac55531a 00007fa8b22e365c 00007fa8b23dd5de 00007fa8b08d64a4 00007fa8b08d6432 00007fa8b08d6361 00007fa8b08d62dc 00007fa8b08d5592 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:43.397338 (+5609190us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:59:41.540219 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:59:41.567596 (+ 27377us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:59:41.567634 (+ 38us) write_op.cc:435] Acquired schema lock
0321 02:59:41.567647 (+ 13us) tablet.cc:661] Decoding operations
0321 02:59:41.570523 (+ 2876us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:59:41.570582 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0321 02:59:41.570621 (+ 39us) tablet.cc:684] Acquiring locks for 17 operations
0321 02:59:41.572885 (+ 2264us) tablet.cc:700] Row locks acquired
0321 02:59:41.572920 (+ 35us) write_op.cc:260] PREPARE: finished
0321 02:59:41.573071 (+ 151us) write_op.cc:270] Start()
0321 02:59:41.573183 (+ 112us) write_op.cc:276] Timestamp: P: 1774061981573020 usec, L: 0
0321 02:59:41.573197 (+ 14us) op_driver.cc:348] REPLICATION: starting
0321 02:59:41.574070 (+ 873us) log.cc:844] Serialized 6634 byte log entry
0321 02:59:41.632947 (+ 58877us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b440002304c. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93c2dc5 00007fa8a93c37e2 00007fa8ac0f757b 00007fa8ac0f2e4f 00007fa8a9498fbc 00007fa8b2276b19 00007fa8a94b546b 00007fa8a94b370f 00007fa8a94b59d2 00007fa8ac0e55ab
0321 02:59:42.027409 (+394462us) spinlock_profiling.cc:243] Waited 922 us on lock 0x7b440002304c. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93c2dc5 00007fa8a93c37e2 00007fa8ac0f757b 00007fa8ac0f2e4f 00007fa8ac0e6c0e 00007fa8ac0e5f56 00007fa8ac0e5b78 00007fa8ac0ef5c1 00007fa8ac0ef579 00007fa8ac0ef509
0321 02:59:42.463878 (+436469us) op_driver.cc:464] REPLICATION: finished
0321 02:59:42.972032 (+508154us) write_op.cc:301] APPLY: starting
0321 02:59:42.976417 (+ 4385us) tablet.cc:1366] starting BulkCheckPresence
0321 02:59:43.022909 (+ 46492us) tablet.cc:1369] finished BulkCheckPresence
0321 02:59:43.025733 (+ 2824us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:59:43.041869 (+ 16136us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:59:43.042166 (+ 297us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=17
0321 02:59:43.042218 (+ 52us) write_op.cc:312] APPLY: finished
0321 02:59:43.042622 (+ 404us) log.cc:844] Serialized 119 byte log entry
0321 02:59:43.042806 (+ 184us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:43.042973 (+ 167us) write_op.cc:454] Released schema lock
0321 02:59:43.043028 (+ 55us) write_op.cc:341] FINISH: Updating metrics
0321 02:59:43.043444 (+ 416us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:43.043476 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":30381,"reported_tablets":44,"spinlock_wait_cycles":2475464576,"child_traces":[["op",{"apply.queue_time_us":455197,"apply.run_cpu_time_us":4386,"apply.run_wall_time_us":89410,"num_ops":17,"prepare.queue_time_us":82416,"prepare.run_cpu_time_us":13384,"prepare.run_wall_time_us":124260,"raft.queue_time_us":224911,"raft.run_cpu_time_us":11659,"raft.run_wall_time_us":279403,"replication_time_us":851858,"spinlock_wait_cycles":34018816,"thread_start_us":407450,"threads_started":4,"wal-append.queue_time_us":80310}]]}
I20260321 02:59:43.467561 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:43.481316 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:43.490396 11236 raft_consensus.cc:1240] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260321 02:59:43.491042 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:43.491761 11385 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:43.492364 11275 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:43.465853 11349 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:43.428908 14343 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:43.498997 11349 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:43.458772 14307 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:43.532452 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 48412.9us
I20260321 02:59:43.549582 14076 raft_consensus.cc:2804] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:43.476855 14296 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
W20260321 02:59:43.551082 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 52211.5us
I20260321 02:59:43.555076 11378 raft_consensus.cc:1240] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 9. Current term is 10. Ops: []
I20260321 02:59:43.550343 14076 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:43.532181 11228 consensus_queue.cc:260] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.600069 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:43.599952 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:43.602023 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 126929us
I20260321 02:59:43.601825 11228 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Advancing to term 11
W20260321 02:59:43.602902 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:43.593901 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 170645us
I20260321 02:59:43.620277 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: true dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:43.621193 11385 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Advancing to term 11
W20260321 02:59:43.637614 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 145316us
I20260321 02:59:43.551888 14307 raft_consensus.cc:3060] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:43.642243 11942 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:43.592504 14118 raft_consensus.cc:2749] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:43.592355 14122 raft_consensus.cc:2804] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:43.498059 14278 consensus_queue.cc:1059] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.130s
I20260321 02:59:43.475694 11973 rpcz_store.cc:276] 0321 02:59:42.266185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:42.278343 (+ 12158us) service_pool.cc:225] Handling call
0321 02:59:42.372067 (+ 93724us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:42.453714 (+ 81647us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:42.457860 (+ 4146us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:42.541762 (+ 83902us) log.cc:844] Serialized 26 byte log entry
0321 02:59:42.699043 (+157281us) raft_consensus.cc:1645] Marking committed up to 7
0321 02:59:42.706428 (+ 7385us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:42.710991 (+ 4563us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:43.250549 (+539558us) spinlock_profiling.cc:243] Waited 6.13 ms on lock 0x7b440002304c. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93c2dc5 00007fa8a93c37e2 00007fa8ac0f757b 00007fa8ac0f2e4f 00007fa8ac0e6c0e 00007fa8ac0e5f56 00007fa8ac0e5b78 00007fa8ac0ef5c1 00007fa8ac0ef579 00007fa8ac0ef509
0321 02:59:43.373905 (+123356us) raft_consensus.cc:1685] finished
0321 02:59:43.375571 (+ 1666us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:43.412292 (+ 36721us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17357,"raft.queue_time_us":29931,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":14547,"spinlock_wait_cycles":14092672,"thread_start_us":113000,"threads_started":1,"wal-append.queue_time_us":208290}
I20260321 02:59:43.612016 14076 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:43.645656 11275 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:43.600458 14098 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:43.527242 14309 consensus_queue.cc:1059] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.107s
I20260321 02:59:43.625197 14343 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:43.669840 11977 raft_consensus.cc:1275] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260321 02:59:43.670373 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 66857.2us
I20260321 02:59:43.669864 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:43.650409 14298 raft_consensus.cc:2749] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:43.679003 14336 raft_consensus.cc:2804] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:43.679726 14336 raft_consensus.cc:697] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:43.680766 14336 consensus_queue.cc:237] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:43.683058 14345 consensus_queue.cc:1059] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.213s
I20260321 02:59:43.685209 14262 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:43.686416 14282 raft_consensus.cc:2804] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:43.686899 14282 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:43.687253 14282 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
W20260321 02:59:43.687927 14350 log.cc:927] Time spent T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260321 02:59:43.690445 11200 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:43.640057 14295 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:43.647763 14257 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:43.647591 14024 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:43.652534 14122 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:43.708603 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:43.709251 11380 raft_consensus.cc:3060] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
W20260321 02:59:43.718030 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:43.724848 11385 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 11.
I20260321 02:59:43.710157 14122 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:43.692960 14343 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:43.775759 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 85392.2us
I20260321 02:59:43.719636 14295 consensus_queue.cc:1048] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.052s
I20260321 02:59:43.712925 14024 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:43.776176 14107 consensus_meta.cc:287] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 1.185s user 0.005s sys 0.009s
I20260321 02:59:43.809861 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:43.815476 14295 consensus_queue.cc:1059] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:59:43.816905 14295 raft_consensus.cc:3055] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Stepping down as leader of term 8
I20260321 02:59:43.817307 14295 raft_consensus.cc:740] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:43.781699 14362 consensus_queue.cc:1059] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.046s
I20260321 02:59:43.773337 14317 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:43.818333 14226 consensus_meta.cc:287] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.597s user 0.007s sys 0.003s
I20260321 02:59:43.818253 14295 consensus_queue.cc:260] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.819049 11232 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:43.794308 14318 raft_consensus.cc:2804] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260321 02:59:43.848234 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:43.851269 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 234974us
I20260321 02:59:43.807258 14074 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:43.819913 14295 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:43.863369 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:43.869129 11377 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:43.865346 11378 raft_consensus.cc:1275] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260321 02:59:43.857645 14309 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:43.874729 11349 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:43.887133 11275 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:43.854421 14318 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:43.889258 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 112322us
W20260321 02:59:43.892657 11349 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:43.825867 14261 consensus_queue.cc:1048] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.063s
I20260321 02:59:43.914325 11200 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:43.862927 13865 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:43.823231 14107 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.888952 14118 raft_consensus.cc:2749] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260321 02:59:43.934338 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 46091.1us
I20260321 02:59:43.928576 11377 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:43.931872 11228 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 11.
I20260321 02:59:43.845712 14363 consensus_queue.cc:1059] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.085s
I20260321 02:59:43.891047 14362 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:43.903484 14318 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:43.941048 11275 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:43.963238 14254 raft_consensus.cc:493] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:43.938112 13664 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:43.974072 14327 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:43.932374 14098 raft_consensus.cc:2804] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election won for term 9
I20260321 02:59:43.923916 14366 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:43.888438 14309 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.914798 14274 consensus_queue.cc:1048] T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.070s
W20260321 02:59:43.996115 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 55186.1us
W20260321 02:59:44.026775 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 90711.6us
I20260321 02:59:43.933538 13865 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.984251 11227 raft_consensus.cc:1240] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260321 02:59:44.030763 11377 raft_consensus.cc:1240] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:44.035578 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:44.040232 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:44.006785 14162 consensus_meta.cc:287] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 1.064s user 0.006s sys 0.007s
W20260321 02:59:44.003249 14357 log.cc:927] Time spent T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.136s user 0.001s sys 0.000s
I20260321 02:59:44.053465 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:44.054296 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:44.056754 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:44.060307 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:44.064404 11229 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:44.065786 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:44.066713 14024 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:44.067564 14024 consensus_queue.cc:1048] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260321 02:59:44.070497 14282 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:43.994707 14278 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:44.073256 11310 raft_consensus.cc:2393] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 8: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:44.044014 11380 raft_consensus.cc:2468] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 8.
I20260321 02:59:44.045408 11275 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:44.077526 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:44.078605 11232 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 10.
I20260321 02:59:44.004477 13933 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 11 FOLLOWER]: Leader election won for term 11
W20260321 02:59:44.080657 11349 leader_election.cc:336] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Timed out: RequestConsensusVote RPC to 127.9.213.1:36717 timed out after 1.821s (SENT)
I20260321 02:59:44.081318 11349 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:43.981799 13846 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:44.005697 14107 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:44.104640 11942 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:44.106590 11275 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:44.106830 11349 leader_election.cc:336] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Timed out: RequestConsensusVote RPC to 127.9.213.1:36717 timed out after 1.844s (SENT)
I20260321 02:59:44.107239 11349 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:44.109927 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:44.103132 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:44.111383 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:44.116535 11228 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 7.
I20260321 02:59:44.118011 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:43.980538 14362 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:44.128225 11349 leader_election.cc:336] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Timed out: RequestConsensusVote RPC to 127.9.213.1:36717 timed out after 1.869s (SENT)
I20260321 02:59:44.146343 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:44.045254 14098 raft_consensus.cc:697] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:44.158700 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:44.159464 11304 raft_consensus.cc:3060] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 8
I20260321 02:59:44.163193 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:44.163295 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:44.176517 11228 raft_consensus.cc:2368] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 6. Current term is 7.
I20260321 02:59:44.156694 11232 raft_consensus.cc:2393] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 8: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:43.995919 13664 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:43.984709 14254 raft_consensus.cc:515] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:44.106213 14024 raft_consensus.cc:2804] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election won for term 8
W20260321 02:59:44.145332 14296 consensus_meta.cc:287] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.525s user 0.011s sys 0.000s
I20260321 02:59:44.120599 13933 raft_consensus.cc:697] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 11 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
W20260321 02:59:44.215524 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 93539.5us
I20260321 02:59:44.216231 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:44.025439 14366 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:44.219559 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:44.157491 14118 raft_consensus.cc:2749] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260321 02:59:44.157925 14255 raft_consensus.cc:2749] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260321 02:59:44.164233 14299 raft_consensus.cc:2804] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:44.229851 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:44.230163 11235 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 8.
I20260321 02:59:44.231231 11308 raft_consensus.cc:2410] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260321 02:59:44.108992 13846 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:44.249655 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:44.105533 14200 consensus_queue.cc:1059] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.108s
W20260321 02:59:44.243063 11201 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:44.102676 14278 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:44.246069 11228 raft_consensus.cc:1240] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260321 02:59:44.065912 14307 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:44.249903 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:44.255316 11231 raft_consensus.cc:2393] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 10: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
W20260321 02:59:44.231514 14374 log.cc:927] Time spent T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260321 02:59:44.269074 11234 raft_consensus.cc:1240] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 9. Current term is 10. Ops: [9.5-9.5]
W20260321 02:59:44.279141 11350 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 67037.1us
I20260321 02:59:44.291643 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:44.250211 14296 raft_consensus.cc:2804] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:44.261641 14343 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:44.270671 14299 raft_consensus.cc:697] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:44.332015 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:44.329887 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:44.329187 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 97443.2us
I20260321 02:59:44.433104 11304 raft_consensus.cc:2468] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 8.
W20260321 02:59:44.402990 14356 log.cc:927] Time spent T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.132s user 0.007s sys 0.000s
W20260321 02:59:44.963485 11276 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.001s sys 0.014s
W20260321 02:59:45.124939 11276 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.719s user 0.003s sys 0.014s
I20260321 02:59:44.404544 14098 consensus_queue.cc:237] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:45.127190 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:45.127975 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:44.443722 11231 raft_consensus.cc:2393] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 9: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:44.404935 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:44.328061 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:44.492007 11349 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:44.963224 11348 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.003s sys 0.012s
W20260321 02:59:45.130358 11348 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.673s user 0.005s sys 0.013s
I20260321 02:59:44.347352 14307 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:44.726524 14384 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:45.003139 11306 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56616 (request call id 1837) took 1583 ms (client timeout 2070 ms). Trace:
I20260321 02:59:45.131556 14384 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:45.131841 11306 rpcz_store.cc:269] 0321 02:59:43.420006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:43.420166 (+ 160us) service_pool.cc:225] Handling call
0321 02:59:45.002874 (+1582708us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b54003a7d54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:45.003020 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39467264}
I20260321 02:59:45.132673 11380 raft_consensus.cc:1240] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:45.134474 14384 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:45.134814 11378 raft_consensus.cc:1275] T 72e86c671bb747c785d8f676ed3b8315 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260321 02:59:44.860736 14387 log.cc:927] Time spent T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.431s user 0.001s sys 0.000s
W20260321 02:59:44.402650 14348 log.cc:927] Time spent T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20260321 02:59:44.347757 14309 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:44.860344 14336 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:45.140411 14336 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:44.402382 14343 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:45.144127 14311 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:45.144622 14343 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:45.144659 14311 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:45.147349 14311 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:44.327821 14024 raft_consensus.cc:697] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:44.426920 14255 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:44.670936 14383 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:44.315754 14296 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:44.999899 14390 ts_tablet_manager.cc:933] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717)
I20260321 02:59:45.147349 14343 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:44.660771 13664 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:45.234599 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:45.005185 14163 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:44.345996 14380 log.cc:927] Time spent T b270af8b9da943d2b91acbdc0bd603cc P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
W20260321 02:59:45.168118 11274 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.009s sys 0.004s
I20260321 02:59:44.408738 14363 raft_consensus.cc:3055] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Stepping down as leader of term 8
W20260321 02:59:44.346154 14076 consensus_meta.cc:287] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.655s user 0.006s sys 0.006s
W20260321 02:59:44.348438 14375 log.cc:927] Time spent T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.130s user 0.000s sys 0.001s
W20260321 02:59:45.063232 14318 consensus_meta.cc:287] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 1.078s user 0.013s sys 0.000s
W20260321 02:59:44.763406 14386 log.cc:927] Time spent T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.447s user 0.003s sys 0.000s
I20260321 02:59:45.206487 14296 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
W20260321 02:59:44.691874 14295 consensus_meta.cc:287] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.806s user 0.012s sys 0.000s
I20260321 02:59:45.204384 11228 raft_consensus.cc:1240] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:44.491148 14282 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:45.006345 14207 raft_consensus.cc:993] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb: : Instructing follower 83b12b883a7f4129a87dadf5a78f08da to start an election
I20260321 02:59:44.401229 14278 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:44.315402 14274 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:44.348279 13933 consensus_queue.cc:237] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.150194 14319 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:45.062806 14323 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:44.891984 14119 raft_consensus.cc:3055] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:44.570459 14395 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:45.280911 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:45.251322 14408 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:44.525650 14254 leader_election.cc:290] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:44.763223 13846 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:44.763691 14397 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:44.350000 13865 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:45.241079 11274 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.592s user 0.009s sys 0.004s
I20260321 02:59:44.345983 14394 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:44.671872 14122 consensus_meta.cc:287] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.902s user 0.010s sys 0.003s
I20260321 02:59:45.164986 14362 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:45.204087 14307 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:45.248833 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:45.250447 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 120299us
I20260321 02:59:45.207280 14309 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:45.236017 13723 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:45.233256 14200 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:45.235775 14392 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:45.262138 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:45.262173 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:45.263563 11311 raft_consensus.cc:1240] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:45.255671 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 8 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:44.402424 14366 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:45.204881 14298 consensus_queue.cc:1059] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260321 02:59:45.242901 14363 raft_consensus.cc:740] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:44.592000 14299 consensus_queue.cc:237] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:45.268678 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:45.257385 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:45.272039 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:45.278519 11379 raft_consensus.cc:1240] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:45.278427 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:45.278821 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:45.278084 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:45.267024 14378 log.cc:927] Time spent T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.737s user 0.001s sys 0.001s
I20260321 02:59:44.777777 14370 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:45.418954 14399 log.cc:927] Time spent T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.128s user 0.004s sys 0.004s
I20260321 02:59:45.249714 14076 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.204926 14255 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.255048 14402 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:45.244246 14163 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:45.267418 14207 raft_consensus.cc:1081] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Signalling peer 83b12b883a7f4129a87dadf5a78f08da to start an election
I20260321 02:59:45.265285 14282 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:45.272761 13664 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:45.450237 13664 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:45.249061 14318 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.452432 13664 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:45.453358 14318 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:45.291864 11302 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49786 (request call id 256) took 2605 ms (client timeout 1777 ms). Trace:
I20260321 02:59:45.430951 14392 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:45.454056 11302 rpcz_store.cc:269] 0321 02:59:42.684722 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:42.684876 (+ 154us) service_pool.cc:225] Handling call
0321 02:59:45.289400 (+2604524us) spinlock_profiling.cc:243] Waited 626 ms on lock 0x7b540031a454. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:45.289531 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1440252800}
I20260321 02:59:45.455039 14384 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:45.455189 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:45.455510 14392 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:45.455926 11302 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 8.
I20260321 02:59:45.438097 14370 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:45.457072 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:45.457170 14392 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:45.328737 14398 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:45.458235 14370 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:45.458007 14398 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:45.312961 14394 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:45.459002 14345 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:45.459528 14345 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:45.460462 14394 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:45.460687 14398 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:45.461365 14398 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:45.463137 14402 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.465523 14402 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:45.465907 14392 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.468636 14392 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:45.293023 14397 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.310676 14122 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.469230 14255 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:45.469895 11382 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:45.471133 14122 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:45.284458 14323 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.473579 14323 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:45.285027 14319 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:45.473940 11381 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40950 (request call id 1397) took 1610 ms. Trace:
I20260321 02:59:45.474372 11381 rpcz_store.cc:276] 0321 02:59:43.863728 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:43.863916 (+ 188us) service_pool.cc:225] Handling call
0321 02:59:43.892044 (+ 28128us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:44.042738 (+150694us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:44.042780 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:44.108036 (+ 65256us) log.cc:844] Serialized 26 byte log entry
0321 02:59:44.314964 (+206928us) raft_consensus.cc:1645] Marking committed up to 3
0321 02:59:44.315023 (+ 59us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:44.317614 (+ 2591us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:45.473704 (+1156090us) raft_consensus.cc:1685] finished
0321 02:59:45.473724 (+ 20us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:45.473881 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12418,"raft.queue_time_us":74747,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":5399,"thread_start_us":206403,"threads_started":2,"wal-append.queue_time_us":1038064}
I20260321 02:59:45.476140 14319 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:45.477171 11305 raft_consensus.cc:3060] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:45.283285 14278 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:45.286422 14395 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.481518 14076 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:45.482311 11383 raft_consensus.cc:1240] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:45.482715 14395 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:45.272209 14274 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.294039 14407 raft_consensus.cc:493] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:45.484606 14274 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:45.405476 11232 raft_consensus.cc:1240] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260321 02:59:45.484330 14407 raft_consensus.cc:515] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:45.434365 14295 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:45.485661 14295 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.487284 14407 leader_election.cc:290] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:45.407039 14162 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:45.485661 14278 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.488261 14162 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.489198 14163 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:45.487843 14295 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:45.491205 14162 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:45.401713 14074 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:45.413939 11308 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56616 (request call id 1843) took 1961 ms (client timeout 2256 ms). Trace:
I20260321 02:59:45.415766 11974 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:45.419739 14309 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:45.492363 11308 rpcz_store.cc:269] 0321 02:59:43.452341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:43.452463 (+ 122us) service_pool.cc:225] Handling call
0321 02:59:45.413699 (+1961236us) spinlock_profiling.cc:243] Waited 741 ms on lock 0x7b540018b854. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:45.413869 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1704207104}
I20260321 02:59:45.493693 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:45.494563 14309 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:45.494845 11308 raft_consensus.cc:2410] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260321 02:59:45.434305 14366 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:45.420157 11385 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40950 (request call id 1395) took 1555 ms. Trace:
I20260321 02:59:45.495530 14366 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.496105 11385 rpcz_store.cc:276] 0321 02:59:43.861657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:43.861818 (+ 161us) service_pool.cc:225] Handling call
0321 02:59:43.887282 (+ 25464us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:43.904806 (+ 17524us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:43.904856 (+ 50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:43.918205 (+ 13349us) log.cc:844] Serialized 26 byte log entry
0321 02:59:44.158105 (+239900us) raft_consensus.cc:1645] Marking committed up to 3
0321 02:59:44.158146 (+ 41us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:44.158703 (+ 557us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:44.762139 (+603436us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b440002304c. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93c2dc5 00007fa8a93c37e2 00007fa8ac0f757b 00007fa8ac0f2e4f 00007fa8ac0e6c0e 00007fa8ac0e5f56 00007fa8ac0e5b78 00007fa8ac0ef5c1 00007fa8ac0ef579 00007fa8ac0ef509
0321 02:59:45.295664 (+533525us) raft_consensus.cc:1685] finished
0321 02:59:45.295689 (+ 25us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:45.416431 (+120742us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16772,"raft.queue_time_us":28499,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":10644,"spinlock_wait_cycles":26798336,"thread_start_us":176389,"threads_started":1,"wal-append.queue_time_us":306616}
I20260321 02:59:45.497738 14366 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:45.497860 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:45.498524 11385 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:45.498827 14363 consensus_queue.cc:260] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:45.420317 11349 leader_election.cc:336] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Timed out: RequestConsensusVote RPC to 127.9.213.1:36717 timed out after 1.844s (SENT)
I20260321 02:59:45.427711 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:45.492435 14074 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:45.500361 11311 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:45.425884 11384 raft_consensus.cc:2393] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 10: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:45.501590 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:45.502074 14074 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:45.278851 14119 raft_consensus.cc:740] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:45.502936 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:45.428123 11377 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:45.503402 14119 consensus_queue.cc:260] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:45.500205 14363 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:45.504707 14119 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:45.506232 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:45.506877 11308 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:45.251631 14336 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:45.507776 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:45.508275 14397 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:45.441715 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:45.439921 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:45.509568 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:45.510398 11309 raft_consensus.cc:2410] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260321 02:59:45.425467 13846 raft_consensus.cc:493] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:45.266410 14024 consensus_queue.cc:237] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.446435 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:45.446321 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:45.447564 11200 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:45.411346 14098 consensus_queue.cc:1059] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.036s
I20260321 02:59:45.449318 14282 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.427608 14200 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:45.455808 14384 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.205684 14383 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.287256 14408 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:45.461452 14345 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:45.476287 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:45.407672 11310 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:49786 (request call id 265) took 1632 ms (client timeout 1957 ms). Trace:
I20260321 02:59:45.491446 14278 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:45.407801 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:45.410154 11230 raft_consensus.cc:1240] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Rejecting Update request from peer 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 5. Current term is 6. Ops: []
I20260321 02:59:45.427227 14403 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:45.500190 11349 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:45.504065 14298 consensus_queue.cc:1048] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260321 02:59:45.508378 11308 raft_consensus.cc:3060] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
W20260321 02:59:45.443795 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 242100us
I20260321 02:59:45.428671 11306 raft_consensus.cc:2410] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260321 02:59:45.425117 14118 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:45.418799 14393 log.cc:927] Time spent T b968fa0c5fb84474b7445627011bf12a P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.164s user 0.006s sys 0.002s
I20260321 02:59:45.420048 14091 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:45.412385 13865 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:45.514780 11305 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:45.515091 11385 raft_consensus.cc:2410] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260321 02:59:45.516438 11302 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 7.
I20260321 02:59:45.519490 11377 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:45.515822 11311 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
I20260321 02:59:45.429620 14406 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:45.523442 11228 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 10.
W20260321 02:59:45.523945 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 76490.2us
I20260321 02:59:45.462065 14398 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:45.522009 11382 raft_consensus.cc:2468] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:45.419075 14307 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.529973 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:45.535586 11276 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:45.753897 11200 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:45.759938 11200 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:45.647456 11275 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:45.643198 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:45.760004 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
W20260321 02:59:45.760954 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 340640us
I20260321 02:59:45.761248 11385 raft_consensus.cc:3060] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 7
I20260321 02:59:45.700613 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:45.642397 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:45.764154 14390 tablet_copy_client.cc:323] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Beginning tablet copy session from remote peer at address 127.9.213.1:36717
W20260321 02:59:45.753741 11310 rpcz_store.cc:269] 0321 02:59:43.775177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:43.778352 (+ 3175us) service_pool.cc:225] Handling call
0321 02:59:45.407464 (+1629112us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b5400319554. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:45.407613 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":598630656}
I20260321 02:59:45.766438 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:45.643795 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:45.767086 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:45.769471 11377 raft_consensus.cc:1275] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260321 02:59:45.767827 14091 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.772996 11384 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:45.776260 14318 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:45.769601 14163 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:45.769755 14309 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:45.784556 14309 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.664480 14366 raft_consensus.cc:493] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:45.785483 14366 raft_consensus.cc:515] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:45.787472 14366 leader_election.cc:290] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:45.784968 11384 raft_consensus.cc:2410] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260321 02:59:45.792338 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:45.795202 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:45.755308 14319 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:45.796061 14416 raft_consensus.cc:2804] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:45.822069 11307 raft_consensus.cc:2410] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260321 02:59:45.666792 14295 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:45.688884 14395 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
W20260321 02:59:45.824535 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 62870.3us
I20260321 02:59:45.664762 14407 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:45.824930 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:45.844327 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:45.755393 14397 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:45.844637 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:45.840083 11378 raft_consensus.cc:1275] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260321 02:59:45.831596 11349 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:45.527094 14162 raft_consensus.cc:2804] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:45.824043 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:45.836246 11377 raft_consensus.cc:1275] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260321 02:59:45.836418 11973 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:45.843135 11385 raft_consensus.cc:2468] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:45.708148 14419 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:45.847916 11381 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:45.767776 11379 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
W20260321 02:59:45.866251 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 330752us
I20260321 02:59:45.868165 11349 leader_election.cc:400] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 6. Current term is 7.
I20260321 02:59:45.868783 11349 leader_election.cc:403] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260321 02:59:45.526690 13846 raft_consensus.cc:515] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:45.837754 14394 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:45.898033 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:45.833837 14076 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:45.877169 11308 raft_consensus.cc:2468] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:45.890540 14397 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:45.899533 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:45.890671 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:45.900192 11349 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:45.889775 11230 raft_consensus.cc:2393] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 9: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:45.834460 14274 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:45.763497 14118 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.838157 14416 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:45.762903 13865 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.900445 11978 raft_consensus.cc:2468] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 3.
I20260321 02:59:45.866822 14407 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.844812 14295 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.901077 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:45.933696 11308 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 7.
I20260321 02:59:45.906876 11349 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:45.841647 14323 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:45.864923 14162 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:45.896217 14315 consensus_queue.cc:1059] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20260321 02:59:45.912853 14091 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:45.909822 14412 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:45.843063 14319 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.765529 14403 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:45.914139 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:45.912701 11351 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.733s user 0.025s sys 0.050s
I20260321 02:59:45.907249 14416 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:45.925323 14408 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:45.914758 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 139810us
I20260321 02:59:45.914853 14327 raft_consensus.cc:2749] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:45.927163 14118 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:45.917423 11236 raft_consensus.cc:1240] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260321 02:59:45.928583 14200 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:45.863368 14395 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:45.988497 11376 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:45.892634 14366 raft_consensus.cc:2804] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:45.905218 14282 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:45.904534 14255 raft_consensus.cc:2749] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for earlier term 6. Current term is 7.
I20260321 02:59:45.890223 14419 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:45.965976 11351 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.994s user 0.026s sys 0.050s
I20260321 02:59:45.914330 14418 raft_consensus.cc:2749] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:45.933419 14281 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:46.005443 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:45.948391 14162 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
W20260321 02:59:45.978724 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 72107.9us
I20260321 02:59:46.006628 11349 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:45.767601 14406 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.981779 11306 raft_consensus.cc:1240] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260321 02:59:45.982599 11304 raft_consensus.cc:1275] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260321 02:59:45.902722 14394 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.902878 14076 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.947078 14307 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:45.984035 11276 leader_election.cc:304] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:45.985883 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:46.015645 11381 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 8.
I20260321 02:59:45.954015 14163 raft_consensus.cc:2749] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260321 02:59:45.989334 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 119088us
I20260321 02:59:46.022250 11200 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:45.907073 14274 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:45.967305 14091 raft_consensus.cc:2749] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260321 02:59:45.976315 14311 raft_consensus.cc:3055] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:46.029006 14311 raft_consensus.cc:740] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:46.029903 14311 consensus_queue.cc:260] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:46.030797 14318 raft_consensus.cc:2749] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:46.031139 14311 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:46.010377 14327 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:45.982134 14098 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:46.031744 14327 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:46.032264 14098 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:46.033870 14327 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:46.034227 14098 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:46.035130 14411 raft_consensus.cc:3055] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 9 LEADER]: Stepping down as leader of term 9
W20260321 02:59:46.024834 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 133602us
I20260321 02:59:46.026095 11200 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:46.004527 14296 consensus_meta.cc:287] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.744s user 0.008s sys 0.003s
I20260321 02:59:45.951999 14323 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:45.992488 14366 raft_consensus.cc:493] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:46.031253 14207 raft_consensus.cc:3055] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Stepping down as leader of term 8
W20260321 02:59:46.049615 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 59665.7us
W20260321 02:59:46.055027 11942 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:46.023655 14384 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:46.058104 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 51568.9us
I20260321 02:59:46.061257 11200 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:46.037092 14418 raft_consensus.cc:2749] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:46.063781 11200 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:46.046590 14307 raft_consensus.cc:2804] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260321 02:59:46.014721 14281 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:46.051239 14415 log.cc:927] Time spent T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
I20260321 02:59:46.045747 13664 raft_consensus.cc:2804] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:46.065241 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:45.966493 14412 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:46.068655 11200 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:46.050690 13846 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:46.073287 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:46.073711 14422 log.cc:927] Time spent T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260321 02:59:46.074273 14163 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:46.044816 14296 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:46.042333 14411 raft_consensus.cc:740] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:46.048056 14366 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:46.071455 14419 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:46.064319 11349 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:46.096486 11275 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:46.453291 14118 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:46.453919 14384 raft_consensus.cc:2749] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260321 02:59:46.454118 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 357755us
I20260321 02:59:46.453969 14118 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:46.455125 11275 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:46.455924 14384 raft_consensus.cc:2804] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260321 02:59:46.456308 14384 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:46.456631 14384 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Advancing to term 11
I20260321 02:59:46.456765 14076 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:46.456947 14118 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:46.393675 14162 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:46.459584 14162 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:46.461001 11276 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:46.461870 14402 raft_consensus.cc:2804] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:46.111694 14304 raft_consensus.cc:2804] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:46.462370 14402 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:46.462728 14304 raft_consensus.cc:697] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:46.462893 14402 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:46.463083 14407 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:46.178385 14271 raft_consensus.cc:993] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb: : Instructing follower 83b12b883a7f4129a87dadf5a78f08da to start an election
I20260321 02:59:46.463716 14304 consensus_queue.cc:237] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:46.464609 14430 raft_consensus.cc:1081] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Signalling peer 83b12b883a7f4129a87dadf5a78f08da to start an election
I20260321 02:59:46.464607 14163 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:46.077862 14328 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:46.466830 14163 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:46.467523 14328 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:46.470084 14328 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:46.049176 14340 raft_consensus.cc:2749] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:46.470532 11304 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1903) took 1190 ms. Trace:
I20260321 02:59:46.076834 14319 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:46.086633 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:46.077513 11234 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58214 (request call id 1442) took 1858 ms. Trace:
I20260321 02:59:46.074965 14395 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:46.194038 11376 raft_consensus.cc:2410] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20260321 02:59:46.472450 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 403864us
I20260321 02:59:46.074782 14318 raft_consensus.cc:2804] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:46.409107 14416 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:46.452713 11942 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:46.058795 14398 leader_election.cc:290] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:46.052301 14207 raft_consensus.cc:740] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:46.596946 11380 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1862) took 1340 ms. Trace:
I20260321 02:59:46.630565 11227 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58214 (request call id 1419) took 3902 ms. Trace:
W20260321 02:59:46.630299 11235 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58200 (request call id 1735) took 2584 ms (client timeout 1746 ms). Trace:
W20260321 02:59:46.865775 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 413155us
I20260321 02:59:46.531662 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:46.505196 11307 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56630 (request call id 1885) took 1243 ms. Trace:
I20260321 02:59:46.557047 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:46.514289 11231 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59202 (request call id 254) took 1106 ms. Trace:
I20260321 02:59:46.882192 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:46.889912 11385 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 8.
I20260321 02:59:46.869583 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:46.868430 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:46.894915 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:46.878631 14281 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:46.831760 14296 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:46.877857 11304 rpcz_store.cc:276] 0321 02:59:45.280389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:46.019159 (+738770us) service_pool.cc:225] Handling call
0321 02:59:46.019458 (+ 299us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:46.019712 (+ 254us) raft_consensus.cc:1534] Early marking committed up to index 4
0321 02:59:46.038185 (+ 18473us) log.cc:844] Serialized 14 byte log entry
0321 02:59:46.470125 (+431940us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:46.470181 (+ 56us) raft_consensus.cc:1645] Marking committed up to 4
0321 02:59:46.470226 (+ 45us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:46.470297 (+ 71us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:46.470452 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":21207,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":54,"raft.queue_time_us":1039,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":9037,"spinlock_wait_cycles":1993088,"thread_start_us":26856,"threads_started":1,"wal-append.queue_time_us":414388}
I20260321 02:59:46.878787 14119 raft_consensus.cc:2804] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:46.512144 14271 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:46.874665 11234 rpcz_store.cc:276] 0321 02:59:44.218581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:44.231308 (+ 12727us) service_pool.cc:225] Handling call
0321 02:59:44.265141 (+ 33833us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:44.269675 (+ 4534us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:46.077285 (+1807610us) spinlock_profiling.cc:243] Waited 8.71 ms on lock 0x7b5400173ed0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54bc8d 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:46.077429 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20032896}
I20260321 02:59:46.866768 11978 raft_consensus.cc:2410] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260321 02:59:46.157580 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:46.880460 11977 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:46.865592 11380 rpcz_store.cc:276] 0321 02:59:45.256775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:45.256938 (+ 163us) service_pool.cc:225] Handling call
0321 02:59:45.257246 (+ 308us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:45.257611 (+ 365us) raft_consensus.cc:1534] Early marking committed up to index 1
0321 02:59:45.257638 (+ 27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:45.266011 (+ 8373us) log.cc:844] Serialized 26 byte log entry
0321 02:59:45.526755 (+260744us) raft_consensus.cc:1645] Marking committed up to 1
0321 02:59:45.526795 (+ 40us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:45.527023 (+ 228us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:46.596637 (+1069614us) raft_consensus.cc:1685] finished
0321 02:59:46.596666 (+ 29us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:46.596869 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":452,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":4795,"thread_start_us":239500,"threads_started":1,"wal-append.queue_time_us":621535}
I20260321 02:59:46.895707 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:46.896329 11385 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
W20260321 02:59:46.861611 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 797278us
I20260321 02:59:46.874238 11227 rpcz_store.cc:276] 0321 02:59:42.718738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:42.718928 (+ 190us) service_pool.cc:225] Handling call
0321 02:59:42.771325 (+ 52397us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:44.029899 (+1258574us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:46.620898 (+2590999us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b540037b254. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:46.621137 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2786719104}
W20260321 02:59:46.879643 11235 rpcz_store.cc:269] 0321 02:59:44.036371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:44.036892 (+ 521us) service_pool.cc:225] Handling call
0321 02:59:46.620385 (+2583493us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b54003336d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:46.620562 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":357618432}
I20260321 02:59:46.913082 11349 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:46.890336 11229 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58214 (request call id 1431) took 3865 ms (client timeout 2249 ms). Trace:
I20260321 02:59:46.663129 11383 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40934 (request call id 1860) took 1407 ms. Trace:
I20260321 02:59:46.913667 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:46.904314 14281 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:46.916389 14271 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:46.917251 11378 raft_consensus.cc:2393] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 9: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:46.881469 14318 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:46.917874 14318 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:46.918586 14271 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:46.061685 14383 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:46.913801 11383 rpcz_store.cc:276] 0321 02:59:45.256125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:45.256258 (+ 133us) service_pool.cc:225] Handling call
0321 02:59:45.275173 (+ 18915us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:45.482746 (+207573us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:46.662773 (+1180027us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b5400373a54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:46.663041 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":475938560}
I20260321 02:59:46.922425 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 11 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:46.922470 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:46.924132 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:46.911744 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:46.924737 11379 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
W20260321 02:59:46.630831 14360 log.cc:927] Time spent T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 2.892s user 0.005s sys 0.000s
W20260321 02:59:46.078447 13723 consensus_meta.cc:287] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.659s user 0.010s sys 0.002s
I20260321 02:59:46.889873 11307 rpcz_store.cc:276] 0321 02:59:45.262122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:45.263442 (+ 1320us) service_pool.cc:225] Handling call
0321 02:59:46.504908 (+1241466us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b540018ae54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:46.505111 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":842713344}
I20260321 02:59:46.501572 14311 raft_consensus.cc:2804] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:46.480225 14295 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:46.914255 11349 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:46.914743 11228 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:46.514596 14384 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:46.831092 14282 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.675s user 0.007s sys 0.007s
I20260321 02:59:46.078222 14394 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:46.904765 14274 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:46.914923 14119 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:46.916715 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:46.904335 14309 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:46.568463 13664 raft_consensus.cc:697] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
W20260321 02:59:46.914224 11229 rpcz_store.cc:269] 0321 02:59:43.025026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:43.030241 (+ 5215us) service_pool.cc:225] Handling call
0321 02:59:46.862276 (+3832035us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540027f1d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:46.890269 (+ 27993us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2363582336}
I20260321 02:59:46.975399 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:46.908902 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:46.920157 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:46.976161 11229 raft_consensus.cc:2393] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 8: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:46.976912 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:46.920291 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:46.921128 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:46.155625 14403 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:46.154582 14397 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:46.075263 14323 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:46.080945 14331 raft_consensus.cc:3055] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Stepping down as leader of term 8
I20260321 02:59:46.151474 14406 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:46.925065 11942 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.9.213.2:34267 timed out after 1.957s (SENT) [suppressed 21 similar messages]
I20260321 02:59:46.630421 14439 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:46.909700 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:46.889451 11231 rpcz_store.cc:276] 0321 02:59:45.408374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:45.408941 (+ 567us) service_pool.cc:225] Handling call
0321 02:59:45.409217 (+ 276us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:45.529690 (+120473us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:45.529728 (+ 38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:45.542620 (+ 12892us) log.cc:844] Serialized 26 byte log entry
0321 02:59:45.950386 (+407766us) raft_consensus.cc:1645] Marking committed up to 7
0321 02:59:46.014393 (+ 64007us) log.cc:844] Serialized 14 byte log entry
0321 02:59:46.026860 (+ 12467us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:46.513857 (+486997us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b540007eed4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:46.513902 (+ 45us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:46.513956 (+ 54us) raft_consensus.cc:1685] finished
0321 02:59:46.514071 (+ 115us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:46.514228 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8332,"prepare.queue_time_us":123382,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":36841,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":104093,"spinlock_wait_cycles":234500352,"thread_start_us":425344,"threads_started":2,"wal-append.queue_time_us":599271}
I20260321 02:59:46.865151 14307 raft_consensus.cc:493] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:46.689910 14076 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:46.500398 14163 raft_consensus.cc:2749] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260321 02:59:46.535459 14411 consensus_queue.cc:260] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:46.931823 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.001624 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:46.932739 11233 raft_consensus.cc:2410] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260321 02:59:46.932659 11378 raft_consensus.cc:1240] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:46.934852 14416 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:47.002885 14340 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:47.009546 14340 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:46.066615 14315 consensus_queue.cc:1048] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.859s
I20260321 02:59:46.938093 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:46.513020 14407 consensus_queue.cc:1059] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.415s
I20260321 02:59:46.939518 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:46.067108 14098 consensus_queue.cc:1048] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.862s
W20260321 02:59:46.937359 14282 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.782s user 0.007s sys 0.008s
I20260321 02:59:46.926731 14383 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:46.974089 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:46.975625 14274 consensus_queue.cc:1059] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:59:46.980578 11380 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
W20260321 02:59:47.026544 11384 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40934 (request call id 1878) took 1596 ms (client timeout 1951 ms). Trace:
I20260321 02:59:46.917291 14281 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:46.982549 11200 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:46.929812 14311 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:46.985584 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:46.900080 14402 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:46.918520 14412 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:46.991014 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 65997.4us
I20260321 02:59:46.992262 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:46.922515 14207 consensus_queue.cc:260] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:47.026965 11384 rpcz_store.cc:269] 0321 02:59:45.425767 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:45.502678 (+ 76911us) service_pool.cc:225] Handling call
0321 02:59:47.022199 (+1519521us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b54001d9a54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:47.022317 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":619850624}
W20260321 02:59:46.984547 14304 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.035s sys 0.045s
I20260321 02:59:47.060389 11304 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:47.061506 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:46.995288 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 66540.7us
I20260321 02:59:46.996227 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:46.985946 14261 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:46.695730 14118 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:47.062083 11234 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:46.975421 13664 consensus_queue.cc:237] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.002346 11231 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:47.003118 11377 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:47.006434 11228 raft_consensus.cc:2468] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:46.991075 14331 raft_consensus.cc:740] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:46.975935 14366 raft_consensus.cc:515] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:46.995725 14307 raft_consensus.cc:3060] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Advancing to term 4
I20260321 02:59:47.013732 11235 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:47.016651 11976 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45098 (request call id 439) took 3543 ms. Trace:
I20260321 02:59:47.014101 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:47.063913 14304 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.597s user 0.035s sys 0.045s
W20260321 02:59:47.020975 11302 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56616 (request call id 1848) took 1583 ms (client timeout 1840 ms). Trace:
W20260321 02:59:47.022723 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:47.069164 14442 log.cc:927] Time spent T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.087s user 0.000s sys 0.001s
I20260321 02:59:47.016987 14384 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 11 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:47.025501 11382 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40934 (request call id 1867) took 1758 ms (client timeout 1718 ms). Trace:
I20260321 02:59:47.026255 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "709aaa09afd74a25bbd5dc62dca37f02" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:47.046430 11232 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58200 (request call id 1749) took 2836 ms. Trace:
I20260321 02:59:47.035285 11227 raft_consensus.cc:2368] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 7. Current term is 8.
W20260321 02:59:47.057473 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 75011.6us
W20260321 02:59:47.057924 11942 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:47.059320 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:46.976373 14119 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Advancing to term 9
W20260321 02:59:47.078610 11382 rpcz_store.cc:269] 0321 02:59:45.267577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:45.268258 (+ 681us) service_pool.cc:225] Handling call
0321 02:59:47.025317 (+1757059us) spinlock_profiling.cc:243] Waited 42 ms on lock 0x7b54001f7cd4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:47.025414 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":96618368}
I20260321 02:59:47.062706 11349 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:47.078989 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:47.087001 11980 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 6.
I20260321 02:59:47.079043 11232 rpcz_store.cc:276] 0321 02:59:44.210086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:44.210261 (+ 175us) service_pool.cc:225] Handling call
0321 02:59:44.216346 (+ 6085us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:45.485300 (+1268954us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:47.046131 (+1560831us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54003b8654. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:47.046356 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2734089984}
I20260321 02:59:47.085536 11942 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:47.000468 11379 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:47.063453 14311 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:47.000305 14076 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.013489 14437 raft_consensus.cc:2749] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260321 02:59:47.068370 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:46.994933 14439 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.069859 11228 raft_consensus.cc:1275] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 11 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260321 02:59:47.089934 11304 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
I20260321 02:59:47.092648 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:47.073187 11307 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:47.019249 11306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1893) took 1753 ms. Trace:
I20260321 02:59:47.075134 11310 raft_consensus.cc:1240] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 8. Current term is 9. Ops: []
W20260321 02:59:47.074780 11302 rpcz_store.cc:269] 0321 02:59:45.437821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:45.456823 (+ 19002us) service_pool.cc:225] Handling call
0321 02:59:47.020812 (+1563989us) spinlock_profiling.cc:243] Waited 58.7 ms on lock 0x7b54003386d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:47.020915 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":135087872}
I20260321 02:59:47.094290 11304 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 8
I20260321 02:59:47.072948 11976 rpcz_store.cc:276] 0321 02:59:43.469401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:43.469581 (+ 180us) service_pool.cc:225] Handling call
0321 02:59:43.474279 (+ 4698us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:43.474605 (+ 326us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:43.474635 (+ 30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:43.475082 (+ 447us) log.cc:844] Serialized 26 byte log entry
0321 02:59:43.593603 (+118521us) raft_consensus.cc:1645] Marking committed up to 7
0321 02:59:43.593668 (+ 65us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:43.593738 (+ 70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:46.630166 (+3036428us) spinlock_profiling.cc:243] Waited 42.7 ms on lock 0x7b440002304c. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93c2dc5 00007fa8a93c37e2 00007fa8ac0f757b 00007fa8ac0f2e4f 00007fa8ac0e6c0e 00007fa8ac0e5f56 00007fa8ac0e5b78 00007fa8ac0ef5c1 00007fa8ac0ef579 00007fa8ac0ef509
0321 02:59:47.000758 (+370592us) raft_consensus.cc:1685] finished
0321 02:59:47.000786 (+ 28us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:47.012130 (+ 11344us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1646,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"spinlock_wait_cycles":98182016,"thread_start_us":117864,"threads_started":1,"wal-append.queue_time_us":214934}
I20260321 02:59:47.064193 14319 raft_consensus.cc:2749] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260321 02:59:47.023525 11311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1892) took 1757 ms. Trace:
I20260321 02:59:47.076745 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.056663 14340 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:47.080530 11200 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:47.069854 14261 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.071677 14207 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:47.069337 14118 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.106751 14304 raft_consensus.cc:2749] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260321 02:59:47.109536 14439 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:47.089308 11235 raft_consensus.cc:1240] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:47.017055 14411 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:47.093711 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:47.092023 14412 raft_consensus.cc:2804] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:47.114323 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.098975 11200 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:47.115058 11303 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56616 (request call id 1842) took 3667 ms (client timeout 2054 ms). Trace:
I20260321 02:59:47.107116 11306 rpcz_store.cc:276] 0321 02:59:45.266013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:45.838979 (+572966us) service_pool.cc:225] Handling call
0321 02:59:45.866549 (+ 27570us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:46.007621 (+141072us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:47.018989 (+1011368us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b54001982d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:47.019196 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":264308480}
I20260321 02:59:47.115832 11200 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:47.121959 11201 leader_election.cc:304] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:47.124331 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 227005us
I20260321 02:59:47.105792 14261 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:47.125151 11275 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:47.085376 11377 raft_consensus.cc:2468] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
I20260321 02:59:47.097002 11311 rpcz_store.cc:276] 0321 02:59:45.265750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:45.766409 (+500659us) service_pool.cc:225] Handling call
0321 02:59:45.766681 (+ 272us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:45.952686 (+186005us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:45.952719 (+ 33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:45.952763 (+ 44us) raft_consensus.cc:1645] Marking committed up to 4
0321 02:59:45.952810 (+ 47us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:47.023373 (+1070563us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b540018bd54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:47.023406 (+ 33us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:47.023486 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":427042176}
I20260321 02:59:47.128432 11306 raft_consensus.cc:1275] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260321 02:59:47.128981 11306 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49786 (request call id 267) took 1281 ms. Trace:
I20260321 02:59:47.129246 11306 rpcz_store.cc:276] 0321 02:59:45.847749 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:47.119281 (+1271532us) service_pool.cc:225] Handling call
0321 02:59:47.128030 (+ 8749us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:47.128808 (+ 778us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:47.128941 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:47.098219 14445 raft_consensus.cc:2749] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:47.102497 11978 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:47.104064 14076 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:47.064952 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.108429 11973 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00eeadd5c566459b86e97f920a91c398" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:47.093117 11942 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:47.111244 11980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:47.113263 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:47.099272 11302 raft_consensus.cc:1240] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:47.117429 14412 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting leader election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:47.117478 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:47.087839 14416 consensus_queue.cc:1059] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260321 02:59:47.114982 11307 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56616 (request call id 1851) took 1670 ms. Trace:
I20260321 02:59:47.138824 11236 raft_consensus.cc:1240] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9. Ops: []
W20260321 02:59:47.115536 11303 rpcz_store.cc:269] 0321 02:59:43.447353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:43.447830 (+ 477us) service_pool.cc:225] Handling call
0321 02:59:45.665619 (+2217789us) spinlock_profiling.cc:243] Waited 6.74 ms on lock 0x7b5800002a98. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a9478e51 00007fa8a9478d02 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b150e760 00007fa8b150e044 00007fa8aaa8f834 00007fa8aaa8f7c2
0321 02:59:47.114161 (+1448542us) spinlock_profiling.cc:243] Waited 3.36 ms on lock 0x7b5800002a98. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a947917c 00007fa8a9478d3b 00007fa8b23193c9 00007fa8b14975e0 00007fa8b156076b 00007fa8b15607f1 00007fa8b150e760 00007fa8b150e044 00007fa8aaa8f834 00007fa8aaa8f7c2
0321 02:59:47.114976 (+ 815us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23227904}
I20260321 02:59:47.145823 11302 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56616 (request call id 1849) took 1702 ms. Trace:
I20260321 02:59:47.132108 11311 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:49786 (request call id 268) took 1256 ms. Trace:
I20260321 02:59:47.146169 11302 rpcz_store.cc:276] 0321 02:59:45.443793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:47.096697 (+1652904us) service_pool.cc:225] Handling call
0321 02:59:47.099152 (+ 2455us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:47.145576 (+ 46424us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:47.145757 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:47.145202 11307 rpcz_store.cc:276] 0321 02:59:45.444526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:47.114165 (+1669639us) service_pool.cc:225] Handling call
0321 02:59:47.114444 (+ 279us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:47.114605 (+ 161us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:47.114629 (+ 24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:47.114668 (+ 39us) raft_consensus.cc:1645] Marking committed up to 7
0321 02:59:47.114712 (+ 44us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:47.114785 (+ 73us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:47.114919 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:47.146970 11311 rpcz_store.cc:276] 0321 02:59:45.872448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:47.128208 (+1255760us) service_pool.cc:225] Handling call
0321 02:59:47.128402 (+ 194us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:47.128514 (+ 112us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:47.128528 (+ 14us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:47.128555 (+ 27us) raft_consensus.cc:1645] Marking committed up to 5
0321 02:59:47.128572 (+ 17us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:47.128613 (+ 41us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:47.128677 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260321 02:59:47.147583 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.148643 11307 raft_consensus.cc:2393] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 8: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:47.148090 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:47.156312 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 63270.7us
I20260321 02:59:47.156848 11311 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 7.
I20260321 02:59:47.157702 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.157785 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:47.158327 14416 raft_consensus.cc:3055] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Stepping down as leader of term 5
I20260321 02:59:47.158700 11235 raft_consensus.cc:2393] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 7: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:47.159689 14419 raft_consensus.cc:2804] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:47.160156 11310 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1906) took 1871 ms. Trace:
I20260321 02:59:47.160290 14419 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:47.159832 11305 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56630 (request call id 1886) took 1897 ms (client timeout 1718 ms). Trace:
I20260321 02:59:47.125226 14340 raft_consensus.cc:2804] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election won for term 9
I20260321 02:59:47.131848 14407 raft_consensus.cc:493] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:47.160584 11310 rpcz_store.cc:276] 0321 02:59:45.288737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:46.022560 (+733823us) service_pool.cc:225] Handling call
0321 02:59:46.023018 (+ 458us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:47.093907 (+1070889us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:47.159923 (+ 66016us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54001d01d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54d1e1 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06
0321 02:59:47.160100 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2419223296}
I20260321 02:59:47.123540 14118 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:47.157090 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:47.160584 11233 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58200 (request call id 1738) took 3114 ms (client timeout 1869 ms). Trace:
I20260321 02:59:47.183815 14281 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:47.184150 11233 rpcz_store.cc:269] 0321 02:59:44.045865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:44.065506 (+ 19641us) service_pool.cc:225] Handling call
0321 02:59:47.160384 (+3094878us) spinlock_profiling.cc:243] Waited 2.87 s on lock 0x7b5400263ed4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:47.160517 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6591119744}
I20260321 02:59:47.148967 14107 consensus_queue.cc:1048] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.036s
I20260321 02:59:47.144587 14261 raft_consensus.cc:2749] T 709aaa09afd74a25bbd5dc62dca37f02 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260321 02:59:47.161273 11305 rpcz_store.cc:269] 0321 02:59:45.262408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:45.263515 (+ 1107us) service_pool.cc:225] Handling call
0321 02:59:47.159652 (+1896137us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b5400319f54. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:47.159752 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":449932672}
I20260321 02:59:47.119727 14304 raft_consensus.cc:2749] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260321 02:59:47.164105 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.096946 14331 consensus_queue.cc:260] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.127343 11304 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:47.187968 14331 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:47.186079 14407 raft_consensus.cc:515] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.189694 11304 raft_consensus.cc:1275] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260321 02:59:47.157807 14366 leader_election.cc:290] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:47.168244 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.179169 11311 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4"
dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:56630
W20260321 02:59:47.191620 11275 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:47.192030 11275 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:47.165678 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.182821 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:47.125231 14383 raft_consensus.cc:2749] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:47.192811 11235 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Advancing to term 11
I20260321 02:59:47.185245 11307 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264"
dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
from {username='slave'} at 127.0.0.1:56630
I20260321 02:59:47.193298 11310 raft_consensus.cc:1275] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260321 02:59:47.189276 14163 raft_consensus.cc:3055] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 8 LEADER]: Stepping down as leader of term 8
I20260321 02:59:47.160806 14419 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:47.149006 14412 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:47.190886 11228 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:47.190857 11309 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56630 (request call id 1891) took 1925 ms. Trace:
I20260321 02:59:47.220781 14402 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:47.248864 11309 rpcz_store.cc:276] 0321 02:59:45.265464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:45.511068 (+245604us) service_pool.cc:225] Handling call
0321 02:59:47.021416 (+1510348us) raft_consensus.cc:1471] Updating replica for 0 ops
0321 02:59:47.021588 (+ 172us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:47.021630 (+ 42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0321 02:59:47.021670 (+ 40us) raft_consensus.cc:1645] Marking committed up to 4
0321 02:59:47.021699 (+ 29us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:47.021763 (+ 64us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:47.190701 (+168938us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54003386d0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac54bc8d 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:47.190794 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2913672064}
I20260321 02:59:47.215310 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:47.185199 14107 consensus_queue.cc:1243] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer 83b12b883a7f4129a87dadf5a78f08da log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
W20260321 02:59:47.242980 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 50496us
I20260321 02:59:47.199776 13865 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:47.222517 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.219903 11132 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44688 (request call id 132) took 6194 ms. Trace:
I20260321 02:59:47.254599 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.255228 11304 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 6.
I20260321 02:59:47.256227 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.257172 11304 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 9.
I20260321 02:59:47.258234 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.253770 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:47.261775 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.244601 14163 raft_consensus.cc:740] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:47.241467 14340 raft_consensus.cc:697] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:47.198307 11307 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
W20260321 02:59:47.208184 14360 log.cc:927] Time spent T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260321 02:59:47.195116 14309 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:47.191232 14293 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:47.158731 14416 raft_consensus.cc:740] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:47.287544 14416 consensus_queue.cc:260] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.288851 14416 raft_consensus.cc:3060] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:47.290453 14311 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.302681 14311 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:47.245745 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.245860 11235 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 11.
I20260321 02:59:47.247761 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:47.250806 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:47.308418 14307 raft_consensus.cc:515] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.310015 11230 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:47.310166 11311 raft_consensus.cc:493] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:59:47.310657 11311 raft_consensus.cc:3060] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 4 FOLLOWER]: Advancing to term 5
I20260321 02:59:47.317966 14307 leader_election.cc:290] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:47.266515 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
W20260321 02:59:47.267025 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 47162.3us
I20260321 02:59:47.336376 11275 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:47.336759 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.337867 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.338979 11309 raft_consensus.cc:2393] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:47.339677 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.340677 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.267941 11307 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
W20260321 02:59:47.204284 11376 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40934 (request call id 1868) took 1925 ms (client timeout 1648 ms). Trace:
W20260321 02:59:47.271116 14443 log.cc:927] Time spent T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.200s user 0.000s sys 0.001s
W20260321 02:59:47.348968 11376 rpcz_store.cc:269] 0321 02:59:45.278422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:45.278600 (+ 178us) service_pool.cc:225] Handling call
0321 02:59:47.204012 (+1925412us) spinlock_profiling.cc:243] Waited 559 ms on lock 0x7b540009a954. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:47.204193 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1286379904}
I20260321 02:59:47.263640 14163 consensus_queue.cc:260] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.350332 14163 raft_consensus.cc:3060] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
W20260321 02:59:47.351092 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 230560us
W20260321 02:59:47.352218 11349 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:47.360687 11307 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.292069 14293 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.362138 14331 raft_consensus.cc:2749] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:47.363247 14293 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:47.292910 14309 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.365351 14309 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:47.365971 11304 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 9.
I20260321 02:59:47.366629 11303 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:47.369460 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:47.370087 11976 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:47.299422 14412 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.255254 14119 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.373602 14119 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 9 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:47.251956 14318 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.374722 14412 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:47.375334 14318 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Requested vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:47.376039 11977 raft_consensus.cc:1240] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 9 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:47.245115 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.376933 11976 raft_consensus.cc:2468] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:47.377338 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.377952 11310 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 7.
I20260321 02:59:47.378181 11276 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:47.378687 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.379012 14331 raft_consensus.cc:2804] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:47.379559 11310 raft_consensus.cc:2393] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 8: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:47.304134 11302 raft_consensus.cc:2393] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 8: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:47.379727 14331 raft_consensus.cc:697] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:47.379833 11303 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 8.
I20260321 02:59:47.319350 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:47.380815 14331 consensus_queue.cc:237] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.249253 14407 leader_election.cc:290] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:47.265895 11200 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:47.383333 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 117522us
I20260321 02:59:47.383503 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.383634 11308 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56616 (request call id 1847) took 1948 ms. Trace:
I20260321 02:59:47.344096 14383 raft_consensus.cc:493] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:47.384678 11304 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 10.
I20260321 02:59:47.384702 11200 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:47.385123 14383 raft_consensus.cc:515] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.385665 11200 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:47.387280 14318 raft_consensus.cc:2749] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260321 02:59:47.344398 11309 raft_consensus.cc:2410] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260321 02:59:47.341506 14307 consensus_queue.cc:1059] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:59:47.387323 14383 leader_election.cc:290] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:47.389070 14309 consensus_queue.cc:1059] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260321 02:59:47.389853 13664 raft_consensus.cc:2804] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:47.390192 14309 raft_consensus.cc:3055] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Stepping down as leader of term 8
I20260321 02:59:47.390398 13664 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:47.390657 14309 raft_consensus.cc:740] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:47.274140 14340 consensus_queue.cc:237] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.391036 13664 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:47.334055 11311 raft_consensus.cc:515] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.395913 11306 raft_consensus.cc:1240] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 3. Current term is 5. Ops: []
I20260321 02:59:47.344125 11275 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:47.403087 14293 raft_consensus.cc:2804] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260321 02:59:47.403651 14293 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:47.404142 14293 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Advancing to term 10
W20260321 02:59:47.408113 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 67121.1us
I20260321 02:59:47.409157 13664 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.409564 11275 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:47.411744 14309 consensus_queue.cc:260] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.414417 14309 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:47.329156 14419 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.429901 14419 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 9 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:47.325424 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 137996us
I20260321 02:59:47.430397 14407 raft_consensus.cc:3055] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:47.431038 14407 raft_consensus.cc:740] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:47.432219 11350 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:47.433913 14383 raft_consensus.cc:2804] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:47.434468 14383 raft_consensus.cc:493] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:47.388203 14318 raft_consensus.cc:2804] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:47.434966 14383 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:47.435104 14318 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:47.435633 14318 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
W20260321 02:59:47.436172 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:47.437125 14407 consensus_queue.cc:260] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.438896 14407 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:47.439471 11376 raft_consensus.cc:1275] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260321 02:59:47.386665 11200 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:47.390738 11307 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
W20260321 02:59:47.440330 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 53768.9us
I20260321 02:59:47.253258 11132 rpcz_store.cc:276] 0321 02:59:41.035519 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:41.038341 (+ 2822us) service_pool.cc:225] Handling call
0321 02:59:47.219814 (+6181473us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:59:43.304243 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:59:43.304817 (+ 574us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:59:43.304851 (+ 34us) write_op.cc:435] Acquired schema lock
0321 02:59:43.304869 (+ 18us) tablet.cc:661] Decoding operations
0321 02:59:43.305261 (+ 392us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:59:43.305342 (+ 81us) write_op.cc:641] Partition lock acquired for write op
0321 02:59:43.305370 (+ 28us) tablet.cc:684] Acquiring locks for 2 operations
0321 02:59:43.305679 (+ 309us) tablet.cc:700] Row locks acquired
0321 02:59:43.305696 (+ 17us) write_op.cc:260] PREPARE: finished
0321 02:59:43.305851 (+ 155us) write_op.cc:270] Start()
0321 02:59:43.305964 (+ 113us) write_op.cc:276] Timestamp: P: 1774061983305797 usec, L: 0
0321 02:59:43.305985 (+ 21us) op_driver.cc:348] REPLICATION: starting
0321 02:59:43.306733 (+ 748us) log.cc:844] Serialized 907 byte log entry
0321 02:59:43.821241 (+514508us) op_driver.cc:464] REPLICATION: finished
0321 02:59:44.355867 (+534626us) write_op.cc:301] APPLY: starting
0321 02:59:44.430942 (+ 75075us) tablet.cc:1366] starting BulkCheckPresence
0321 02:59:44.510955 (+ 80013us) tablet.cc:1369] finished BulkCheckPresence
0321 02:59:44.540623 (+ 29668us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:59:44.567482 (+ 26859us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:59:44.567720 (+ 238us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0321 02:59:44.567798 (+ 78us) write_op.cc:312] APPLY: finished
0321 02:59:44.568076 (+ 278us) log.cc:844] Serialized 29 byte log entry
0321 02:59:44.568299 (+ 223us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:44.568408 (+ 109us) write_op.cc:454] Released schema lock
0321 02:59:44.568463 (+ 55us) write_op.cc:341] FINISH: Updating metrics
0321 02:59:44.568909 (+ 446us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:44.568950 (+ 41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":13988,"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":471551,"apply.run_cpu_time_us":2776,"apply.run_wall_time_us":241328,"num_ops":2,"prepare.queue_time_us":92863,"prepare.run_cpu_time_us":2821,"prepare.run_wall_time_us":2821,"replication_time_us":466430,"thread_start_us":324843,"threads_started":2}]]}
W20260321 02:59:47.320400 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 157354us
I20260321 02:59:47.424289 14411 raft_consensus.cc:2804] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260321 02:59:47.442452 14411 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:47.442858 14411 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:47.444360 14311 consensus_queue.cc:1059] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260321 02:59:47.445308 14311 raft_consensus.cc:3055] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Stepping down as leader of term 8
I20260321 02:59:47.445654 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:47.446341 11383 raft_consensus.cc:3060] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:47.442780 11200 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:47.448417 14315 raft_consensus.cc:2804] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:47.448586 11200 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:47.449182 14315 raft_consensus.cc:697] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:47.450024 14315 consensus_queue.cc:237] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.451644 14261 raft_consensus.cc:2804] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:47.452174 14261 raft_consensus.cc:697] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
W20260321 02:59:47.452502 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:47.452653 14302 consensus_queue.cc:1048] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260321 02:59:47.452982 14261 consensus_queue.cc:237] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.453526 14027 consensus_queue.cc:1059] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260321 02:59:47.454103 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:47.454551 14302 raft_consensus.cc:3055] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Stepping down as leader of term 3
I20260321 02:59:47.454648 11977 raft_consensus.cc:3060] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 3 FOLLOWER]: Advancing to term 5
I20260321 02:59:47.442842 14207 raft_consensus.cc:2804] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:47.455286 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:47.455467 14207 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:47.456015 14207 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:47.456192 11230 raft_consensus.cc:2410] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260321 02:59:47.448315 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:47.459105 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.452649 11311 leader_election.cc:290] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:47.462949 11977 raft_consensus.cc:2468] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:47.463467 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.445647 14311 raft_consensus.cc:740] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:47.464218 11276 leader_election.cc:304] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:47.464967 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.465445 14307 raft_consensus.cc:2804] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 FOLLOWER]: Leader election won for term 5
I20260321 02:59:47.465080 14311 consensus_queue.cc:260] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.466246 14307 raft_consensus.cc:697] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:47.466840 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:47.467291 14307 consensus_queue.cc:237] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.455016 14302 raft_consensus.cc:740] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:47.469477 14411 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.461594 11201 leader_election.cc:304] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:47.427381 11275 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:47.473068 14384 raft_consensus.cc:2804] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:47.473609 14384 raft_consensus.cc:697] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:47.474679 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.474480 14384 consensus_queue.cc:237] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.475894 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.476280 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.476380 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:47.476953 11379 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:47.477183 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.477305 13723 raft_consensus.cc:2749] T 00eeadd5c566459b86e97f920a91c398 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:47.477917 11234 raft_consensus.cc:2393] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 8: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:47.478726 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.478834 11201 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:47.479434 11233 raft_consensus.cc:2393] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:47.473613 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.480722 14340 raft_consensus.cc:2749] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:47.481423 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:47.472968 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 45740.7us
I20260321 02:59:47.474712 14302 consensus_queue.cc:260] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.483206 11275 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:47.483615 14302 raft_consensus.cc:3060] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 3 FOLLOWER]: Advancing to term 5
I20260321 02:59:47.484127 13865 raft_consensus.cc:2804] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:47.484548 13865 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:47.484903 13865 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
W20260321 02:59:47.488520 14448 log.cc:927] Time spent T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
I20260321 02:59:47.447374 14383 raft_consensus.cc:515] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.466753 14311 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:47.493789 14119 consensus_queue.cc:1048] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260321 02:59:47.496587 11275 leader_election.cc:400] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 7. Current term is 8.
I20260321 02:59:47.496932 11275 leader_election.cc:403] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260321 02:59:47.498428 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:47.498994 11974 raft_consensus.cc:3060] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:47.504480 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
W20260321 02:59:47.378736 11349 connection.cc:232] RPC call timeout handler was delayed by -1.1805420379999987s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18400000000000002s for a response
I20260321 02:59:47.506273 11349 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:47.506575 11974 raft_consensus.cc:2468] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 7.
I20260321 02:59:47.508154 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.507108 11308 rpcz_store.cc:276] 0321 02:59:45.434985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:46.881271 (+1446286us) service_pool.cc:225] Handling call
0321 02:59:46.916819 (+ 35548us) raft_consensus.cc:1471] Updating replica for 1 ops
0321 02:59:46.917180 (+ 361us) raft_consensus.cc:1534] Early marking committed up to index 0
0321 02:59:46.917212 (+ 32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0321 02:59:46.917686 (+ 474us) log.cc:844] Serialized 26 byte log entry
0321 02:59:47.381253 (+463567us) spinlock_profiling.cc:243] Waited 44.9 ms on lock 0x7b540005c420. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b6470 00007fa8ac4c3a98 00007fa8ac54cd0b 00007fa8ac54bc1e 00007fa8b150cad9 00007fa8aaa8ad34 00007fa8aaa8acc2 00007fa8aaa8abf1 00007fa8aaa8ab6c 00007fa8aaa89e22 00007fa8aa8895cc
0321 02:59:47.381312 (+ 59us) raft_consensus.cc:1645] Marking committed up to 8
0321 02:59:47.381603 (+ 291us) log.cc:844] Serialized 14 byte log entry
0321 02:59:47.383243 (+ 1640us) raft_consensus.cc:1697] Filling consensus response to leader.
0321 02:59:47.383315 (+ 72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0321 02:59:47.383357 (+ 42us) raft_consensus.cc:1685] finished
0321 02:59:47.383455 (+ 98us) raft_consensus.cc:1691] UpdateReplicas() finished
0321 02:59:47.383579 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1506,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":1613,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":167,"spinlock_wait_cycles":103196032,"thread_start_us":401563,"threads_started":1,"wal-append.queue_time_us":392469}
I20260321 02:59:47.509907 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:47.510398 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.514089 14402 raft_consensus.cc:2749] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 7. Current term is 8.
I20260321 02:59:47.516247 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.516701 11349 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:47.517035 11228 raft_consensus.cc:2393] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 9: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:47.517637 11379 raft_consensus.cc:2468] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 9.
I20260321 02:59:47.519366 11134 catalog_manager.cc:5671] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 3 to 9. New cstate: current_term: 9 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:47.525043 11349 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:47.525913 11201 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:47.529098 11134 catalog_manager.cc:5671] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:47.529772 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:47.530315 11376 raft_consensus.cc:3060] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 5 FOLLOWER]: Advancing to term 6
I20260321 02:59:47.513944 11383 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 9.
I20260321 02:59:47.494378 13865 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.503360 14318 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.626237 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.629571 11379 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
W20260321 02:59:47.629999 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 95692.7us
W20260321 02:59:47.631562 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:47.632463 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.625715 14163 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:47.634649 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:47.645699 11379 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 10.
I20260321 02:59:47.646327 11974 raft_consensus.cc:2393] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:47.639461 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.647760 11228 raft_consensus.cc:2393] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 9: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:47.530507 11134 catalog_manager.cc:5671] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:47.471812 14411 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:47.524369 14437 raft_consensus.cc:2804] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:47.643630 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:47.644820 11977 raft_consensus.cc:1240] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:47.646531 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:47.524667 14383 leader_election.cc:290] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:47.423447 14293 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.652375 11381 raft_consensus.cc:2393] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 8: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
I20260321 02:59:47.652787 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.653013 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.654654 11383 raft_consensus.cc:2393] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 9: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:47.654646 11379 raft_consensus.cc:2410] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260321 02:59:47.630817 14451 raft_consensus.cc:2804] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election won for term 7
I20260321 02:59:47.642359 11942 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:47.637583 14207 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.653462 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.662847 11228 raft_consensus.cc:2393] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 7: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:47.659446 11275 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:47.660773 11379 raft_consensus.cc:1240] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 9. Current term is 10. Ops: []
I20260321 02:59:47.661875 14412 raft_consensus.cc:2804] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 9 FOLLOWER]: Leader election won for term 9
I20260321 02:59:47.630151 14405 consensus_queue.cc:1059] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.032s
I20260321 02:59:47.665302 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.538636 14394 consensus_queue.cc:1048] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.120s
I20260321 02:59:47.639696 14439 raft_consensus.cc:2804] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:47.660679 11942 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:47.668135 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.676944 11228 raft_consensus.cc:2393] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 8: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:47.662775 11376 raft_consensus.cc:2468] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:47.678102 11379 raft_consensus.cc:1240] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:47.669972 11134 catalog_manager.cc:5671] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:47.651608 14437 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:47.669183 11275 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:47.661693 14261 consensus_queue.cc:1059] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
I20260321 02:59:47.684886 14261 raft_consensus.cc:3055] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Stepping down as leader of term 6
I20260321 02:59:47.685254 14261 raft_consensus.cc:740] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:47.686098 14261 consensus_queue.cc:260] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.681169 14437 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:47.646701 14119 consensus_queue.cc:1048] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20260321 02:59:47.688694 11275 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:47.680303 11134 catalog_manager.cc:5671] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 7 to 8, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 8 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:47.668682 14281 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:47.690497 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:47.693765 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.689702 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.692082 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.694522 11379 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:47.694553 11228 raft_consensus.cc:2393] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 8: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:47.692081 11134 catalog_manager.cc:5671] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:47.655339 14407 consensus_queue.cc:1059] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260321 02:59:47.696135 11134 catalog_manager.cc:5671] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } } }
I20260321 02:59:47.697386 11228 raft_consensus.cc:1275] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260321 02:59:47.697772 11228 pending_rounds.cc:85] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb: Aborting all ops after (but not including) 3
I20260321 02:59:47.697890 11383 raft_consensus.cc:1240] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:47.671111 14384 raft_consensus.cc:2804] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260321 02:59:47.698592 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.698019 11228 pending_rounds.cc:107] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260321 02:59:47.701113 11228 raft_consensus.cc:2889] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:47.694341 11381 raft_consensus.cc:2468] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 6.
I20260321 02:59:47.667045 14451 raft_consensus.cc:697] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:47.700368 11942 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:47.669565 14412 raft_consensus.cc:697] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 9 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:47.705145 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:47.705435 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.705904 11228 raft_consensus.cc:2368] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 4. Current term is 5.
I20260321 02:59:47.705994 11235 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:47.672672 14439 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:47.701632 11134 catalog_manager.cc:5671] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 8 to 9, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 9 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:47.712096 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:47.712704 11236 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:47.706846 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:47.693451 14405 raft_consensus.cc:2804] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 11 FOLLOWER]: Leader election won for term 11
I20260321 02:59:47.687253 14449 raft_consensus.cc:2804] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 9 FOLLOWER]: Leader election won for term 9
I20260321 02:59:47.709143 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.693593 14331 raft_consensus.cc:2804] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:47.717351 11942 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:47.709386 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: true dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:47.722255 14293 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:47.726660 11246 tablet_copy_service.cc:140] P 0d7efc590a1041a9bd4cd4a971332dfb: Received BeginTabletCopySession request for tablet 8f4ec8d11b7b4e6096d8c2b8ece078dc from peer 3d8d3176d6ec407ea2a965556c2e2c56 ({username='slave'} at 127.0.0.1:59202)
I20260321 02:59:47.727444 11246 tablet_copy_service.cc:161] P 0d7efc590a1041a9bd4cd4a971332dfb: Beginning new tablet copy session on tablet 8f4ec8d11b7b4e6096d8c2b8ece078dc from peer 3d8d3176d6ec407ea2a965556c2e2c56 at {username='slave'} at 127.0.0.1:59202: session id = 3d8d3176d6ec407ea2a965556c2e2c56-8f4ec8d11b7b4e6096d8c2b8ece078dc
I20260321 02:59:47.727675 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.707031 14383 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:47.713644 11134 catalog_manager.cc:5671] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:47.711831 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:47.700529 14281 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.720404 14299 consensus_queue.cc:1048] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260321 02:59:47.707401 14261 raft_consensus.cc:3060] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:47.721396 14405 raft_consensus.cc:697] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 11 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:47.721657 14402 raft_consensus.cc:3055] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Stepping down as leader of term 8
I20260321 02:59:47.722079 14411 raft_consensus.cc:3055] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Stepping down as leader of term 8
I20260321 02:59:47.716904 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.705021 14384 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:47.735422 14398 raft_consensus.cc:2749] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:47.738956 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.713894 14439 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:47.740379 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 11 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.766513 14207 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:47.767916 11381 raft_consensus.cc:2410] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 11 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 3.
I20260321 02:59:47.731853 11134 catalog_manager.cc:5671] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 4 to 5, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 5 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:47.771308 14281 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:47.770479 11134 catalog_manager.cc:5671] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 7 to 8, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 8 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:47.724464 14331 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:47.714052 14298 raft_consensus.cc:2804] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:47.776075 14298 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:47.776669 14298 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:47.756698 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4337508daeb4d598c4f6a0b4314878d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.780833 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.782902 11230 raft_consensus.cc:2468] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 5.
I20260321 02:59:47.763757 11234 raft_consensus.cc:2468] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:47.764040 14384 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:47.764437 14451 consensus_queue.cc:237] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.744289 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.736882 14449 raft_consensus.cc:697] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 9 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:47.773149 11134 catalog_manager.cc:5671] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:47.780223 14331 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:47.752115 14411 raft_consensus.cc:740] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:47.752259 13664 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:47.741714 14383 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.787652 11134 catalog_manager.cc:5671] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 7 to 8, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 8 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:47.790494 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:47.780429 14412 consensus_queue.cc:237] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.798552 14449 consensus_queue.cc:237] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.788441 14318 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:47.750314 14402 raft_consensus.cc:740] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:47.803663 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e8870ff9214a65af6cd9f677bffab4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.787246 13865 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:47.805446 11381 raft_consensus.cc:3060] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Advancing to term 7
I20260321 02:59:47.806752 11134 catalog_manager.cc:5671] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 4 to 9, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 9 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:47.813246 11379 raft_consensus.cc:2393] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 8: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:47.804006 11977 raft_consensus.cc:1275] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260321 02:59:47.815254 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:47.816404 11246 tablet_copy_source_session.cc:215] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb: Tablet Copy: opened 0 blocks and 1 log segments
I20260321 02:59:47.828699 14407 consensus_queue.cc:1048] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260321 02:59:47.820627 14318 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:47.830384 14318 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.832350 14318 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:47.837272 14383 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:47.837801 14451 consensus_queue.cc:1048] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:47.837991 14411 consensus_queue.cc:260] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.839337 14411 raft_consensus.cc:3060] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:47.841226 14207 consensus_queue.cc:1059] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:59:47.842648 14207 raft_consensus.cc:3055] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:47.843046 14207 raft_consensus.cc:740] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:47.835314 14384 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.844250 14207 consensus_queue.cc:260] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.845398 14207 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:47.845691 14384 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:47.849947 11236 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 10.
I20260321 02:59:47.850329 11134 catalog_manager.cc:5671] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:47.823870 14437 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:47.817230 14402 consensus_queue.cc:260] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:47.852751 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 149993us
I20260321 02:59:47.851918 11381 raft_consensus.cc:2468] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:47.856175 11200 leader_election.cc:304] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:47.856745 11275 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:47.857559 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:47.851665 11134 catalog_manager.cc:5671] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:47.858619 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:47.859190 11311 raft_consensus.cc:2376] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 8. Re-sending same reply.
I20260321 02:59:47.859127 11134 catalog_manager.cc:5671] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 2 to 7. New cstate: current_term: 7 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:47.860862 11134 catalog_manager.cc:5671] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 6 to 7, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 7 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:47.862639 11134 catalog_manager.cc:5671] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 2 to 7, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 7 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:47.853228 11228 raft_consensus.cc:2468] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:47.863795 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:47.855302 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.864354 13664 raft_consensus.cc:2749] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:47.864615 11311 raft_consensus.cc:2393] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 8: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:47.864418 11134 catalog_manager.cc:5671] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da reported cstate change: term changed from 7 to 8, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2). New cstate: current_term: 8 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } }
I20260321 02:59:47.866650 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:47.868775 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
W20260321 02:59:47.870625 14457 log.cc:927] Time spent T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.146s user 0.001s sys 0.000s
I20260321 02:59:47.866698 11200 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:47.876231 11200 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:47.877660 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.877816 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.878986 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.879622 11384 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:47.879904 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b701d6135bc458c9aef1d5303344525" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.880105 14318 raft_consensus.cc:2749] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260321 02:59:47.880502 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b270af8b9da943d2b91acbdc0bd603cc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.880939 11378 raft_consensus.cc:1275] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 11 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260321 02:59:47.881044 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.882776 11385 raft_consensus.cc:1240] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:47.882920 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.883692 11381 raft_consensus.cc:2393] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 9: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:47.884258 11200 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:47.884410 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.884864 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:47.885365 11236 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:47.885946 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.886868 11380 raft_consensus.cc:2368] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 7. Current term is 8.
I20260321 02:59:47.890357 11236 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 10.
I20260321 02:59:47.891793 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:47.893050 11976 raft_consensus.cc:1240] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 9 LEADER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:47.893329 11236 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:47.895251 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.898095 11309 raft_consensus.cc:1275] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260321 02:59:47.898887 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.900413 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:47.901984 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:47.902529 11377 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:47.904377 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.904534 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.905393 11383 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:47.906106 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.906245 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:47.906829 11384 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 7.
I20260321 02:59:47.907088 14298 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.907723 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.908308 11384 raft_consensus.cc:2468] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:47.908367 14163 raft_consensus.cc:2749] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260321 02:59:47.812822 14405 consensus_queue.cc:237] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.893441 14402 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:47.918272 11379 raft_consensus.cc:1275] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260321 02:59:47.899119 14318 raft_consensus.cc:2804] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election won for term 9
I20260321 02:59:47.919384 14318 raft_consensus.cc:697] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:47.908557 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.914678 11376 raft_consensus.cc:1240] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:47.920836 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.920986 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.908960 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.921453 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:47.922025 11376 raft_consensus.cc:3060] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:47.920746 11381 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 11.
I20260321 02:59:47.922446 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.919711 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.922951 11311 raft_consensus.cc:2468] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 9.
I20260321 02:59:47.923424 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:47.922053 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.924014 11380 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Advancing to term 11
I20260321 02:59:47.924438 11382 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:47.906965 11383 raft_consensus.cc:2393] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 7: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:47.921774 11385 raft_consensus.cc:2410] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260321 02:59:47.921725 11378 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:47.927752 11379 raft_consensus.cc:2393] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 8: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:47.928392 11378 raft_consensus.cc:1240] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:47.899029 14342 raft_consensus.cc:2749] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260321 02:59:47.931521 14437 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:47.934576 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "72e86c671bb747c785d8f676ed3b8315" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.920241 14318 consensus_queue.cc:237] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.935652 14298 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:47.939574 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:47.940100 11977 raft_consensus.cc:1240] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:47.941418 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:47.941978 11378 raft_consensus.cc:3060] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:47.943676 11376 raft_consensus.cc:2468] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 10.
I20260321 02:59:47.916560 14331 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.946065 14302 consensus_queue.cc:1059] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260321 02:59:47.946926 14302 consensus_queue.cc:1243] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer 3d8d3176d6ec407ea2a965556c2e2c56 log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260321 02:59:47.946961 14311 consensus_queue.cc:1059] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260321 02:59:47.947582 14302 raft_consensus.cc:3055] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Stepping down as leader of term 8
I20260321 02:59:47.948138 14302 raft_consensus.cc:740] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:47.913542 14439 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:47.949187 14302 consensus_queue.cc:260] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:47.950513 14302 raft_consensus.cc:3060] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:47.950774 14439 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:47.948220 14311 raft_consensus.cc:3055] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:47.957518 11236 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:47.946311 14331 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:47.962440 11377 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
I20260321 02:59:47.966539 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:47.995847 11311 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:47.995834 11384 raft_consensus.cc:1240] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:47.967343 11376 raft_consensus.cc:1275] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260321 02:59:47.980474 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:47.997840 11376 pending_rounds.cc:85] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5: Aborting all ops after (but not including) 5
I20260321 02:59:47.998131 11376 pending_rounds.cc:107] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260321 02:59:47.998360 11376 raft_consensus.cc:2889] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:47.998201 11379 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
I20260321 02:59:47.987133 11228 raft_consensus.cc:1275] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260321 02:59:47.989910 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:48.000324 11235 raft_consensus.cc:2368] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 7. Current term is 8.
I20260321 02:59:47.955857 14311 raft_consensus.cc:740] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:48.003742 11378 raft_consensus.cc:2468] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 10.
W20260321 02:59:48.027747 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:48.033432 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:48.039530 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:48.058079 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 106178us
I20260321 02:59:48.094187 11976 raft_consensus.cc:1275] T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 5 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260321 02:59:48.102003 14331 consensus_queue.cc:1048] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260321 02:59:48.122184 11384 raft_consensus.cc:1240] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260321 02:59:48.123075 11380 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 11.
W20260321 02:59:48.124286 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:48.127748 11311 raft_consensus.cc:1240] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:48.127929 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:48.132670 11311 pending_rounds.cc:85] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da: Aborting all ops after (but not including) 3
I20260321 02:59:48.133105 11311 pending_rounds.cc:107] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260321 02:59:48.133435 11311 raft_consensus.cc:2889] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:48.141957 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:48.142241 11275 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:48.142647 11307 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:48.143047 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:48.143711 11380 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:48.143942 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:48.145102 11378 raft_consensus.cc:2393] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 7: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:48.146353 11304 raft_consensus.cc:1240] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [term 7 LEADER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 6. Current term is 7. Ops: []
I20260321 02:59:48.149331 11310 raft_consensus.cc:1240] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:48.154798 11275 leader_election.cc:400] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 4. Current term is 5.
I20260321 02:59:48.155176 11275 leader_election.cc:403] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260321 02:59:48.158213 11275 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:48.159209 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:48.160069 11378 raft_consensus.cc:2368] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 9. Current term is 10.
I20260321 02:59:48.159875 11275 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:48.153643 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:48.161823 11310 raft_consensus.cc:2368] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 9. Current term is 10.
I20260321 02:59:48.156526 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:48.162421 11302 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:48.163419 11380 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
I20260321 02:59:48.165656 11380 raft_consensus.cc:1275] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260321 02:59:48.170070 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:48.170692 11310 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:48.172734 14331 raft_consensus.cc:2804] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election won for term 9
I20260321 02:59:48.173316 14331 raft_consensus.cc:697] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [term 9 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:48.172739 11275 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:48.174147 14331 consensus_queue.cc:237] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:48.174786 11306 raft_consensus.cc:1275] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260321 02:59:48.175134 11306 pending_rounds.cc:85] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da: Aborting all ops after (but not including) 3
I20260321 02:59:48.174978 11275 leader_election.cc:304] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:48.176344 14307 raft_consensus.cc:2804] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader election won for term 10
I20260321 02:59:48.176795 11275 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:48.177459 14394 raft_consensus.cc:2804] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader election won for term 10
I20260321 02:59:48.175410 11306 pending_rounds.cc:107] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260321 02:59:48.176867 14307 raft_consensus.cc:697] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 10 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:48.178086 11306 raft_consensus.cc:2889] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:48.191707 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:48.192580 11378 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:48.195333 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:48.196051 11381 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:48.178355 14407 consensus_queue.cc:1048] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20260321 02:59:48.195847 14384 consensus_queue.cc:1048] T 1146889bd4dc49e9a2b67700208bc264 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260321 02:59:48.200601 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:48.202419 11942 leader_election.cc:304] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:48.156891 14411 raft_consensus.cc:2749] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260321 02:59:48.205802 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:48.162025 14309 raft_consensus.cc:2749] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Leader election lost for term 4. Reason: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 4. Current term is 5.
I20260321 02:59:48.219135 11310 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:48.189643 11307 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
W20260321 02:59:48.213934 11200 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:48.178611 14366 raft_consensus.cc:2804] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260321 02:59:48.222373 14366 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:48.222824 14366 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:48.225024 14342 consensus_queue.cc:1048] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:59:48.228046 11200 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:48.229902 14342 raft_consensus.cc:2804] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260321 02:59:48.231562 14342 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:48.232087 14394 raft_consensus.cc:697] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [term 10 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:48.232431 14342 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:48.230594 14315 consensus_queue.cc:1059] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260321 02:59:48.233268 14394 consensus_queue.cc:237] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:48.234062 14207 raft_consensus.cc:3055] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:48.234642 14207 raft_consensus.cc:740] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:48.235761 14207 consensus_queue.cc:260] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:48.238037 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:48.238579 11380 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:48.200506 13865 raft_consensus.cc:2804] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election won for term 9
I20260321 02:59:48.245963 11309 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
I20260321 02:59:48.250433 11378 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:48.253072 11380 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
W20260321 02:59:48.261718 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 103313us
I20260321 02:59:48.156107 14311 consensus_queue.cc:260] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:48.246745 14119 raft_consensus.cc:2749] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260321 02:59:48.262346 11349 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:48.266875 11349 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:48.268275 11349 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:48.269469 11200 leader_election.cc:400] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Vote denied by peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267) with higher term. Message: Invalid argument: T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 9. Current term is 10.
I20260321 02:59:48.270135 11349 leader_election.cc:304] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:48.270495 11200 leader_election.cc:403] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260321 02:59:48.237432 14366 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:48.275379 11349 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:48.260962 13865 raft_consensus.cc:697] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [term 9 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:48.277343 11349 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:48.268484 14439 raft_consensus.cc:2749] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260321 02:59:48.273499 14163 raft_consensus.cc:2749] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260321 02:59:48.282411 13865 consensus_queue.cc:237] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:48.284024 14390 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f4ec8d11b7b4e6096d8c2b8ece078dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260321 02:59:48.279134 14281 raft_consensus.cc:2749] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:48.271617 13664 raft_consensus.cc:2749] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267) with higher term. Message: Invalid argument: T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 9. Current term is 10.
I20260321 02:59:48.274689 14311 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:48.269901 14466 raft_consensus.cc:2749] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260321 02:59:48.272200 14318 consensus_queue.cc:1175] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "83b12b883a7f4129a87dadf5a78f08da" responder_term: 7 status { last_received { term: 7 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 7 index: 7 } }
I20260321 02:59:48.304411 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:48.296137 14439 raft_consensus.cc:2764] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260321 02:59:48.266672 14307 consensus_queue.cc:237] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:48.328074 11276 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:48.295238 11380 raft_consensus.cc:1275] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260321 02:59:48.298785 11977 raft_consensus.cc:1275] T 62b43ed6f6404b7aa8463e168f98232a P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260321 02:59:48.339455 11302 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:48.343904 14119 consensus_queue.cc:1048] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.009s
I20260321 02:59:48.353240 14331 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:48.353847 14331 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:48.354795 14207 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:48.359937 11200 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:48.361997 11200 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:48.362848 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 185021us
I20260321 02:59:48.363679 11275 leader_election.cc:400] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 7. Current term is 8.
I20260321 02:59:48.363999 11275 leader_election.cc:403] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260321 02:59:48.330485 13664 consensus_queue.cc:1048] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260321 02:59:48.361516 13723 raft_consensus.cc:2804] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:48.375514 11200 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:48.377789 11942 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:48.367503 14394 raft_consensus.cc:2749] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717) with higher term. Message: Invalid argument: T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 7. Current term is 8.
I20260321 02:59:48.360313 14261 consensus_queue.cc:1048] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
W20260321 02:59:48.377311 14486 log.cc:927] Time spent T 00000000000000000000000000000000 P c3992e3c917740db9e9d9c810eddf87c: Append to log took a long time: real 0.146s user 0.003s sys 0.000s
I20260321 02:59:48.377854 14331 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:48.394809 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 117510us
I20260321 02:59:48.402434 14412 raft_consensus.cc:2749] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260321 02:59:48.403496 14484 log.cc:927] Time spent T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.062s user 0.000s sys 0.001s
I20260321 02:59:48.401909 11200 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:48.413154 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:48.414011 11234 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:48.389261 14366 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:48.380785 14416 raft_consensus.cc:2804] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election won for term 9
I20260321 02:59:48.410439 11349 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:48.411578 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:48.417568 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:48.418195 11302 raft_consensus.cc:1275] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260321 02:59:48.428320 11349 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:48.418934 11236 raft_consensus.cc:1275] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260321 02:59:48.429633 11349 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:48.405508 11942 leader_election.cc:304] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:48.420084 11378 raft_consensus.cc:1240] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:48.421203 11377 raft_consensus.cc:1275] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260321 02:59:48.426209 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:48.424518 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:48.388798 13723 raft_consensus.cc:697] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:48.427759 11228 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 10.
I20260321 02:59:48.435492 14466 raft_consensus.cc:2749] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260321 02:59:48.436030 14439 raft_consensus.cc:2804] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260321 02:59:48.436544 14439 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:48.437103 14439 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:48.431228 13664 raft_consensus.cc:2955] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 LEADER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:59:48.440285 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:48.434741 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 51909.5us
I20260321 02:59:48.442261 11234 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 10.
I20260321 02:59:48.443881 11349 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:48.442204 11275 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:48.447360 11378 raft_consensus.cc:1240] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:48.450450 11309 raft_consensus.cc:1275] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260321 02:59:48.452924 11378 raft_consensus.cc:1275] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260321 02:59:48.466876 11275 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:48.467866 11275 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:48.480496 11275 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:48.481105 11349 leader_election.cc:304] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:48.482409 11349 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:48.484126 11275 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:48.458168 14449 raft_consensus.cc:2749] T 1146889bd4dc49e9a2b67700208bc264 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:48.488175 11349 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
W20260321 02:59:48.492455 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 86989.5us
I20260321 02:59:48.457823 14163 raft_consensus.cc:493] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:48.502229 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:48.508935 11275 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:48.528517 11275 leader_election.cc:304] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:48.501268 14491 log.cc:927] Time spent T d42ec1d798814a50bb535a7012a76eb4 P 3d8d3176d6ec407ea2a965556c2e2c56: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20260321 02:59:48.486894 14495 raft_consensus.cc:2804] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election won for term 9
I20260321 02:59:48.496565 14340 raft_consensus.cc:2804] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:48.496942 14411 raft_consensus.cc:2764] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260321 02:59:48.533401 11234 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2d71c397418a46d085c9974230b7f964"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:48.524855 14309 raft_consensus.cc:2804] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260321 02:59:48.536320 11275 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:48.499150 14492 raft_consensus.cc:2749] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:48.544561 11234 raft_consensus.cc:606] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:48.559706 11200 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:48.498009 14437 raft_consensus.cc:2749] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:48.502676 14466 raft_consensus.cc:2749] T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260321 02:59:48.521772 14281 raft_consensus.cc:2764] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260321 02:59:48.558137 11275 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:48.497428 14383 raft_consensus.cc:2749] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:48.578626 11275 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:48.580365 11200 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:48.502454 14488 raft_consensus.cc:2804] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader election won for term 6
I20260321 02:59:48.586297 14488 raft_consensus.cc:697] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [term 6 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:48.588259 11234 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:48.588527 11275 leader_election.cc:304] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:48.588871 11234 raft_consensus.cc:606] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:48.589210 11234 raft_consensus.cc:614] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:48.589109 13865 raft_consensus.cc:2749] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:48.529663 14416 raft_consensus.cc:697] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:48.590315 13865 raft_consensus.cc:2749] T 2b701d6135bc458c9aef1d5303344525 P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:48.590796 14416 consensus_queue.cc:237] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:48.591174 13865 consensus_queue.cc:1048] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260321 02:59:48.593637 14493 log.cc:927] Time spent T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.184s user 0.001s sys 0.000s
I20260321 02:59:48.587344 14488 consensus_queue.cc:237] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:48.589999 11275 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:48.596042 11275 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:48.525105 14163 raft_consensus.cc:515] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:48.600036 11275 leader_election.cc:400] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Vote denied by peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533) with higher term. Message: Invalid argument: T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 7. Current term is 8.
I20260321 02:59:48.545464 14494 raft_consensus.cc:2749] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:48.547909 14309 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:48.609479 14309 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:48.610112 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:48.610334 14163 leader_election.cc:290] T 3c50eb057ef548839834ecb7dfd20ce5 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:48.547158 14495 raft_consensus.cc:697] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:48.607005 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:48.572520 14340 raft_consensus.cc:697] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:48.614522 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:48.608264 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62b43ed6f6404b7aa8463e168f98232a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:48.608572 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:48.615751 11230 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 10.
I20260321 02:59:48.572111 14394 raft_consensus.cc:2749] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:48.611359 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:48.612174 11200 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:48.612286 11229 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:48.672448 11228 raft_consensus.cc:2393] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 9: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:48.598363 14411 raft_consensus.cc:2749] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:48.648896 11230 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:48.665875 11233 raft_consensus.cc:1240] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260321 02:59:48.675968 11230 raft_consensus.cc:606] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:48.603068 14331 raft_consensus.cc:2749] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:48.676483 11230 raft_consensus.cc:614] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:48.532775 13723 consensus_queue.cc:237] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:48.603456 14405 raft_consensus.cc:2749] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260321 02:59:48.671170 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 165961us
W20260321 02:59:48.626458 11274 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:48.673074 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 61014us
I20260321 02:59:48.673465 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:48.665414 14293 raft_consensus.cc:2749] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260321 02:59:48.681269 11378 raft_consensus.cc:1275] T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260321 02:59:48.668289 14490 log.cc:927] Time spent T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.226s user 0.002s sys 0.000s
W20260321 02:59:48.664970 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 176865us
I20260321 02:59:48.601240 11275 leader_election.cc:304] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:48.681936 11231 raft_consensus.cc:2393] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 9: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:48.685194 11200 leader_election.cc:304] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:48.687227 11349 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:48.693181 11977 raft_consensus.cc:1240] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Rejecting Update request from peer fef7f8cc55b043039131bd9b2229f0a5 for earlier term 7. Current term is 8. Ops: []
W20260321 02:59:48.696691 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 95551.2us
I20260321 02:59:48.700747 11200 leader_election.cc:400] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Vote denied by peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533) with higher term. Message: Invalid argument: T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 9. Current term is 10.
I20260321 02:59:48.701617 11275 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:48.703022 11275 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:48.703311 11200 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:48.709199 11349 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:48.707224 11275 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:48.701931 14402 raft_consensus.cc:2749] T b270af8b9da943d2b91acbdc0bd603cc P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:48.683072 14511 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260321 02:59:48.712280 11349 leader_election.cc:304] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:48.705133 14451 raft_consensus.cc:2749] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:48.720165 11275 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:48.706462 14500 consensus_queue.cc:1048] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.015s
I20260321 02:59:48.721701 13664 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:59200 (request call id 13) took 13868 ms. Trace:
I20260321 02:59:48.720916 11349 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:48.724998 11231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:48.706936 14407 raft_consensus.cc:2749] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:48.724964 11349 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:48.725734 11231 raft_consensus.cc:606] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:48.708352 14405 raft_consensus.cc:2749] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:48.726166 11231 raft_consensus.cc:614] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:48.712774 14499 raft_consensus.cc:2804] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:48.712587 14281 raft_consensus.cc:2764] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260321 02:59:48.713109 14163 raft_consensus.cc:2749] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:48.730856 11275 leader_election.cc:304] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:48.732038 14163 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260321 02:59:48.730530 14298 consensus_queue.cc:1048] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.002s
W20260321 02:59:48.732666 14505 log.cc:927] Time spent T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.136s user 0.000s sys 0.000s
W20260321 02:59:48.733548 14496 log.cc:927] Time spent T 1b8d0e26d25e4888bc95c0956520bdd1 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.085s user 0.000s sys 0.001s
W20260321 02:59:48.734228 14502 log.cc:927] Time spent T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20260321 02:59:48.727690 14495 consensus_queue.cc:237] T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:48.734946 11231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:48.684584 14508 consensus_queue.cc:1048] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260321 02:59:48.736153 14492 consensus_queue.cc:1059] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260321 02:59:48.722106 13664 rpcz_store.cc:276] 0321 02:59:34.849701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:34.931804 (+ 82103us) service_pool.cc:225] Handling call
0321 02:59:34.973730 (+ 41926us) log.cc:844] Serialized 492 byte log entry
0321 02:59:36.442533 (+1468803us) spinlock_profiling.cc:243] Waited 35.5 ms on lock 0x7b180020de80. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8b2300441 00007fa8b230cca1 00007fa8a94b62f2 00007fa8a94b8591 00007fa8a94b8549 00007fa8a94b84d9 00007fa8a94b84a1 00007fa8a94b779d 0000000000379254 0000000000379099 00007fa8a949974a
0321 02:59:36.693272 (+250739us) spinlock_profiling.cc:243] Waited 8.22 ms on lock 0x7b54001255a0. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac4ccf06 00007fa8ac4c687a 00007fa8ac4c5af4 00007fa8ac4a628a 00007fa8ac4ad133 00007fa8ac4ad0b9 00007fa8ac4ad049 00007fa8ac4ad011 00007fa8ac4ac1dd 0000000000379254
0321 02:59:48.712571 (+12019299us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13253,"raft.queue_time_us":35342,"raft.run_cpu_time_us":32213,"raft.run_wall_time_us":1720987,"spinlock_wait_cycles":100532352,"thread_start_us":369902,"threads_started":2,"wal-append.queue_time_us":379835,"wal-append.run_cpu_time_us":3093,"wal-append.run_wall_time_us":1149478}
I20260321 02:59:48.718550 13865 raft_consensus.cc:2804] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:48.738682 11275 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:48.724473 14511 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 11 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:48.684995 14340 consensus_queue.cc:237] T 6f453202f8fb47abaa1cc6c0fc390553 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:48.793890 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 68992.7us
I20260321 02:59:48.852348 11349 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:48.725081 14411 consensus_queue.cc:1048] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
I20260321 02:59:48.729827 14499 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:48.742455 14405 raft_consensus.cc:2749] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:48.741281 14163 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:48.725565 14383 raft_consensus.cc:2749] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260321 02:59:48.799675 14509 log.cc:927] Time spent T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20260321 02:59:48.726137 14437 raft_consensus.cc:2749] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:48.739543 14402 raft_consensus.cc:2749] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260321 02:59:48.841612 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 103027us
I20260321 02:59:48.903735 11275 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:48.882872 14207 consensus_meta.cc:287] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.528s user 0.011s sys 0.002s
I20260321 02:59:48.873373 14488 raft_consensus.cc:2804] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:48.905651 11275 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:48.861840 14513 log.cc:927] Time spent T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.130s user 0.004s sys 0.000s
W20260321 02:59:48.862946 14342 consensus_meta.cc:287] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.609s user 0.014s sys 0.010s
I20260321 02:59:48.842964 13865 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:48.869153 14511 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 11 FOLLOWER]: Advancing to term 12
W20260321 02:59:48.883791 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 173725us
I20260321 02:59:48.916044 13865 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:48.920176 14302 consensus_queue.cc:1048] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260321 02:59:48.920717 14522 consensus_queue.cc:1059] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:59:48.921592 14261 consensus_queue.cc:1059] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260321 02:59:48.921725 14522 consensus_queue.cc:1048] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:59:48.923115 14302 raft_consensus.cc:3055] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:48.923481 14302 raft_consensus.cc:740] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
W20260321 02:59:48.880515 14311 consensus_meta.cc:287] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb: Time spent flushing consensus metadata: real 0.556s user 0.003s sys 0.005s
I20260321 02:59:48.858075 11349 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:48.924551 11942 leader_election.cc:304] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:48.924854 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 66874.6us
I20260321 02:59:48.908134 11275 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:48.882099 14499 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:48.883713 14163 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Advancing to term 11
I20260321 02:59:48.897866 14492 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:48.915993 11231 raft_consensus.cc:606] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
W20260321 02:59:48.948894 11349 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:48.867703 14520 raft_consensus.cc:2804] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:48.954278 11303 raft_consensus.cc:1275] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260321 02:59:48.930133 11309 raft_consensus.cc:2955] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:59:48.943325 11311 raft_consensus.cc:1275] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260321 02:59:48.942126 11307 raft_consensus.cc:1275] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260321 02:59:48.956180 11381 raft_consensus.cc:1275] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260321 02:59:48.928205 11378 raft_consensus.cc:1275] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260321 02:59:48.957172 11378 pending_rounds.cc:85] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5: Aborting all ops after (but not including) 4
I20260321 02:59:48.957731 11378 pending_rounds.cc:107] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260321 02:59:48.958132 11378 raft_consensus.cc:2889] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:48.910251 14384 raft_consensus.cc:2749] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:48.952996 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:48.928402 11227 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 8.
I20260321 02:59:48.950042 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:48.962883 11384 raft_consensus.cc:3060] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
W20260321 02:59:48.952829 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 44794.8us
I20260321 02:59:48.963667 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:48.964323 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:48.965832 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
W20260321 02:59:48.948684 14439 consensus_meta.cc:287] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5: Time spent flushing consensus metadata: real 0.511s user 0.011s sys 0.001s
I20260321 02:59:48.966087 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:48.944964 14522 raft_consensus.cc:3055] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Stepping down as leader of term 8
I20260321 02:59:48.919541 14309 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:48.967218 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:48.971376 11379 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 10.
I20260321 02:59:48.965435 11376 raft_consensus.cc:1275] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 11 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260321 02:59:48.976229 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:48.968860 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:48.977255 11383 raft_consensus.cc:2368] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 7. Current term is 8.
I20260321 02:59:48.981731 11384 raft_consensus.cc:2468] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:48.966511 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e1ea01442f4d13a0f6432ea463c5df" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:48.985512 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:48.987032 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:48.949494 14383 raft_consensus.cc:2749] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:48.971712 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:48.995321 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:48.995550 11303 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 9.
I20260321 02:59:48.953733 14512 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:48.988942 11235 raft_consensus.cc:1275] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 11 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260321 02:59:48.909574 14488 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:48.990312 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:48.996723 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:48.999610 11303 raft_consensus.cc:1275] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260321 02:59:48.991729 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:48.957618 14307 raft_consensus.cc:2749] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260321 02:59:48.993208 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:48.993619 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:49.031016 11305 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 10.
I20260321 02:59:48.994637 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:48.961210 14281 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:48.913347 14405 raft_consensus.cc:2749] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:48.991576 11384 raft_consensus.cc:1275] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260321 02:59:48.965456 11349 leader_election.cc:304] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:48.973301 14522 raft_consensus.cc:740] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:49.037217 11235 raft_consensus.cc:1275] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260321 02:59:48.925657 14342 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.039585 13664 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:49.040269 13664 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.006590 11378 raft_consensus.cc:2393] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 9: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
W20260321 02:59:49.034228 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 68876.6us
I20260321 02:59:49.042690 11232 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 8.
I20260321 02:59:49.043740 11349 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:49.047554 11235 raft_consensus.cc:1275] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260321 02:59:49.029630 14315 raft_consensus.cc:993] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb: : Instructing follower fef7f8cc55b043039131bd9b2229f0a5 to start an election
I20260321 02:59:49.051424 13664 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:48.961122 14492 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.054603 14309 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:49.027595 14488 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:49.058228 11349 leader_election.cc:304] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:48.975282 14439 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:49.064134 11235 raft_consensus.cc:1275] T e4337508daeb4d598c4f6a0b4314878d P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260321 02:59:49.044993 14394 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:49.044984 14383 raft_consensus.cc:2749] T 006e9dd81c0c4368bed549c97128f4ea P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:49.070009 11349 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:48.975551 14302 consensus_queue.cc:260] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.071447 11349 leader_election.cc:304] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:49.074151 13664 consensus_queue.cc:1048] T e626ec524e8c40ec8062293d8268e85f P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
W20260321 02:59:49.072042 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 121795us
I20260321 02:59:49.072979 11308 raft_consensus.cc:1240] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 9. Current term is 10. Ops: []
I20260321 02:59:49.075899 11233 raft_consensus.cc:1275] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260321 02:59:49.078666 11381 raft_consensus.cc:2393] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 9: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:49.060966 14437 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260321 02:59:49.040836 14281 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.066569 14261 raft_consensus.cc:1081] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Signalling peer fef7f8cc55b043039131bd9b2229f0a5 to start an election
I20260321 02:59:49.067126 14412 raft_consensus.cc:2749] T a42977f759be4144a5862197a77e395d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260321 02:59:49.083770 11376 raft_consensus.cc:1240] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260321 02:59:49.083853 14437 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:49.029696 14512 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.090828 11275 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:49.095034 11977 raft_consensus.cc:3060] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:49.097584 11377 raft_consensus.cc:2468] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:49.098246 11275 leader_election.cc:304] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:49.085573 14383 raft_consensus.cc:2749] T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260321 02:59:49.085521 14309 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:49.100245 11275 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:49.070848 14507 consensus_queue.cc:1048] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260321 02:59:49.073024 14394 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.128198 11977 raft_consensus.cc:1275] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 [term 9 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260321 02:59:49.091609 14437 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Advancing to term 11
I20260321 02:59:49.099269 14302 raft_consensus.cc:3060] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:49.136175 13664 consensus_queue.cc:1059] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260321 02:59:49.112290 14293 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260321 02:59:49.137535 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 66171.2us
I20260321 02:59:49.139552 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:49.140237 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aab7e035f7984ed8ba6560e1591ec831" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:49.140575 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:49.140290 11378 raft_consensus.cc:3060] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:49.114871 14307 raft_consensus.cc:2804] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader election won for term 10
W20260321 02:59:49.129827 14526 log.cc:927] Time spent T 2fcb8dea530d4005a0101183a89b709d P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.052s user 0.000s sys 0.001s
I20260321 02:59:49.144843 11379 raft_consensus.cc:1275] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 7 index: 3. (index mismatch)
I20260321 02:59:49.115880 14520 raft_consensus.cc:697] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:49.146204 11385 raft_consensus.cc:1275] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260321 02:59:49.146823 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260321 02:59:49.150211 14528 log.cc:927] Time spent T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260321 02:59:49.152849 14492 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:49.128247 14331 consensus_queue.cc:1048] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260321 02:59:49.144090 14293 raft_consensus.cc:493] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:49.123945 14366 raft_consensus.cc:2804] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election won for term 9
I20260321 02:59:49.091444 14405 consensus_queue.cc:1048] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260321 02:59:49.156673 11378 raft_consensus.cc:2468] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 10.
I20260321 02:59:49.091789 14497 consensus_queue.cc:1048] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.046s
I20260321 02:59:49.095427 14311 consensus_queue.cc:1048] T 446eeb6a12c64deb848b6db18a6587f6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260321 02:59:49.144948 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:49.160893 11379 raft_consensus.cc:1275] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260321 02:59:49.098270 14207 consensus_queue.cc:1048] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260321 02:59:49.203898 14511 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.186717 14293 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Advancing to term 11
I20260321 02:59:49.206277 14511 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 12 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:49.206964 14499 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.208240 14439 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:49.143751 14411 consensus_queue.cc:1048] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260321 02:59:49.210929 13865 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.133162 14383 raft_consensus.cc:2749] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260321 02:59:49.073243 14522 consensus_queue.cc:260] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:49.180876 14492 raft_consensus.cc:2749] T 70e1ea01442f4d13a0f6432ea463c5df P fef7f8cc55b043039131bd9b2229f0a5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260321 02:59:49.182767 14512 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:49.182817 14342 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 10 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:49.213047 13865 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:49.213898 14366 raft_consensus.cc:697] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [term 9 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:49.159854 14416 raft_consensus.cc:3055] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Stepping down as leader of term 9
I20260321 02:59:49.214404 14416 raft_consensus.cc:740] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:49.214877 14366 consensus_queue.cc:237] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.215262 14416 consensus_queue.cc:260] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.216245 14416 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:49.125942 14309 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.216809 14520 consensus_queue.cc:237] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.168068 14466 raft_consensus.cc:3055] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:49.128979 14494 consensus_queue.cc:1048] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260321 02:59:49.197479 14394 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:49.198158 11410 heartbeater.cc:646] Failed to heartbeat to 127.9.213.62:36307 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.9.213.62:36307 timed out after 15.000s (SENT)
I20260321 02:59:49.212096 11306 raft_consensus.cc:1240] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 7. Current term is 8. Ops: []
I20260321 02:59:49.218124 14402 consensus_queue.cc:1048] T e4337508daeb4d598c4f6a0b4314878d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260321 02:59:49.218822 14309 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:49.167119 14307 raft_consensus.cc:697] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [term 10 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:49.221966 11230 raft_consensus.cc:1240] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 12 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 11. Current term is 12. Ops: []
I20260321 02:59:49.195725 11384 raft_consensus.cc:2376] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 11. Re-sending same reply.
W20260321 02:59:49.225137 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 95110.5us
I20260321 02:59:49.194972 14451 raft_consensus.cc:493] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:49.240451 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 67363.9us
I20260321 02:59:49.223912 14466 raft_consensus.cc:740] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:49.254271 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:49.259055 14307 consensus_queue.cc:237] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.261037 14163 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.263141 11378 raft_consensus.cc:1240] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 11 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 10. Current term is 11. Ops: []
W20260321 02:59:49.281490 11199 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:49.328258 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:49.328272 11942 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:49.330253 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:49.330895 11378 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 11.
I20260321 02:59:49.331069 11305 raft_consensus.cc:1275] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260321 02:59:49.337276 11378 raft_consensus.cc:2955] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Committing config change with OpId 4.4: config changed from index -1 to 4, NON_VOTER 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } } }
I20260321 02:59:49.272169 14163 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 11 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:49.357453 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:49.347851 14119 raft_consensus.cc:2804] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260321 02:59:49.348884 11377 raft_consensus.cc:1275] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260321 02:59:49.355634 14503 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.004s sys 0.009s
I20260321 02:59:49.364873 10068 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260321 02:59:49.247514 14451 raft_consensus.cc:515] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:49.358347 11304 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 10
I20260321 02:59:49.256572 14531 consensus_queue.cc:1048] T 2fcb8dea530d4005a0101183a89b709d P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.087s
I20260321 02:59:49.366920 10068 auto_leader_rebalancer.cc:451] All tables' leader rebalancing finished this round
I20260321 02:59:49.362764 11377 raft_consensus.cc:1275] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260321 02:59:49.364629 14503 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.639s user 0.005s sys 0.009s
I20260321 02:59:49.364043 14119 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:49.273089 14466 consensus_queue.cc:260] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:49.474145 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:49.424407 11377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:49.479148 11377 raft_consensus.cc:3060] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Advancing to term 10
W20260321 02:59:49.492206 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 372442us
I20260321 02:59:49.455207 11380 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2d71c397418a46d085c9974230b7f964"
dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:40934
I20260321 02:59:49.504545 11380 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20260321 02:59:49.504952 11380 raft_consensus.cc:3060] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:49.506700 11304 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 10.
W20260321 02:59:49.454514 14538 log.cc:927] Time spent T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20260321 02:59:49.512988 11275 leader_election.cc:304] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:49.519351 11377 raft_consensus.cc:2468] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 10.
I20260321 02:59:49.471774 14499 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:49.450701 14513 log.cc:927] Time spent T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.078s user 0.001s sys 0.000s
I20260321 02:59:49.495922 14119 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:49.373363 14488 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.454093 14437 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.551359 11275 leader_election.cc:304] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:49.553689 14494 raft_consensus.cc:2749] T 3c50eb057ef548839834ecb7dfd20ce5 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260321 02:59:49.553959 14546 log.cc:927] Time spent T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
I20260321 02:59:49.555619 14494 consensus_queue.cc:1048] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260321 02:59:49.556440 14521 log.cc:927] Time spent T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.662s user 0.000s sys 0.003s
I20260321 02:59:49.547262 14309 raft_consensus.cc:493] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:49.557263 14309 raft_consensus.cc:515] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:49.557618 11275 leader_election.cc:400] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 election: Vote denied by peer fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533) with higher term. Message: Invalid argument: T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 7. Current term is 8.
I20260321 02:59:49.559341 14309 leader_election.cc:290] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:49.559346 11275 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:49.564395 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:49.549721 14517 raft_consensus.cc:2749] T 72e86c671bb747c785d8f676ed3b8315 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260321 02:59:49.595466 14488 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:49.602290 11380 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.607791 14119 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:49.589550 11309 raft_consensus.cc:1240] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Rejecting Update request from peer 0d7efc590a1041a9bd4cd4a971332dfb for earlier term 8. Current term is 9. Ops: []
I20260321 02:59:49.618185 11380 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:49.612382 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 64564us
I20260321 02:59:49.589346 11275 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:49.525027 14405 consensus_queue.cc:1048] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.049s
I20260321 02:59:49.590507 14530 raft_consensus.cc:2749] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260321 02:59:49.621933 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 45169.1us
I20260321 02:59:49.625440 14437 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 11 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:49.621065 14488 raft_consensus.cc:2749] T aab7e035f7984ed8ba6560e1591ec831 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260321 02:59:49.632220 11275 leader_election.cc:304] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:49.633708 14407 raft_consensus.cc:2749] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:49.634261 14407 consensus_queue.cc:1048] T e8b6fd0841c644779c26179d18ddf697 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260321 02:59:49.636381 14407 consensus_queue.cc:1048] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:59:49.638474 14293 raft_consensus.cc:515] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.639151 14407 consensus_queue.cc:1048] T 562af69310cc422e8b378a98d1bb279b P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260321 02:59:49.640524 14451 leader_election.cc:290] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:49.640636 11275 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:49.644876 11349 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:49.646054 11349 leader_election.cc:304] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:49.647231 11349 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:49.652103 11349 leader_election.cc:304] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:49.654266 11349 leader_election.cc:304] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:49.657424 11349 leader_election.cc:304] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:49.654744 14541 raft_consensus.cc:2749] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260321 02:59:49.649950 14390 consensus_meta.cc:287] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 3d8d3176d6ec407ea2a965556c2e2c56: Time spent flushing consensus metadata: real 0.586s user 0.005s sys 0.006s
I20260321 02:59:49.643457 14405 raft_consensus.cc:2764] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260321 02:59:49.661072 11349 leader_election.cc:304] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:49.662077 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:49.662396 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:49.654001 14520 raft_consensus.cc:2804] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:49.648880 14512 raft_consensus.cc:2749] T aab7e035f7984ed8ba6560e1591ec831 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260321 02:59:49.663354 14545 raft_consensus.cc:2749] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260321 02:59:49.665156 14541 raft_consensus.cc:2749] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260321 02:59:49.665503 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:49.662375 14555 consensus_queue.cc:1048] T 1b8d0e26d25e4888bc95c0956520bdd1 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.009s
I20260321 02:59:49.662791 14163 raft_consensus.cc:2749] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:49.668385 14520 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:49.668838 14520 raft_consensus.cc:3060] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:49.669577 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: true dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:49.670357 11305 raft_consensus.cc:2393] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 10: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:49.670876 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:49.671327 14119 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:49.671409 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:49.672276 11305 raft_consensus.cc:2393] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 11: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:49.654345 14488 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:49.673671 14488 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:49.659672 14437 raft_consensus.cc:2749] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260321 02:59:49.675827 14488 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:49.684409 14522 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:49.687332 11974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:49.688102 11974 raft_consensus.cc:2393] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 8: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
I20260321 02:59:49.689991 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:49.690971 11309 raft_consensus.cc:2393] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 in current term 8: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:49.698804 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:49.699429 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:49.699635 11382 raft_consensus.cc:2393] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 10: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:49.700403 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:49.700563 11376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b8d0e26d25e4888bc95c0956520bdd1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:49.701045 11383 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:49.702042 11377 raft_consensus.cc:1275] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260321 02:59:49.703264 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:49.705973 11275 leader_election.cc:304] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:49.700829 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:49.707224 11275 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:49.707216 11381 raft_consensus.cc:3060] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:49.711412 11384 raft_consensus.cc:1275] T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260321 02:59:49.692011 14507 raft_consensus.cc:493] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:49.692313 14437 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:49.694057 14207 raft_consensus.cc:493] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:49.734691 14411 raft_consensus.cc:2804] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:49.751384 14390 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:45078 (request call id 587) took 5722 ms. Trace:
I20260321 02:59:49.752074 14390 rpcz_store.cc:276] 0321 02:59:44.028711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:44.032542 (+ 3831us) service_pool.cc:225] Handling call
0321 02:59:45.240248 (+1207706us) ts_tablet_manager.cc:934] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 3d8d3176d6ec407ea2a965556c2e2c56: Initiating tablet copy from peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717)
0321 02:59:48.265402 (+3025154us) tablet_copy_client.cc:368] Tablet copy session begun
0321 02:59:48.949683 (+684281us) tablet_metadata.cc:641] Metadata flushed
0321 02:59:48.992324 (+ 42641us) tablet_copy_client.cc:497] Wrote new tablet metadata
0321 02:59:49.706108 (+713784us) tablet_copy_client.cc:418] Wrote new consensus metadata
0321 02:59:49.750598 (+ 44490us) tablet_replica.cc:206] Creating consensus instance
0321 02:59:49.751142 (+ 544us) ts_tablet_manager.cc:989] Replica registered.
0321 02:59:49.751244 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":316,"tablet-copy.queue_time_us":417965,"thread_start_us":472042,"threads_started":2}
I20260321 02:59:49.754829 14520 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.760851 14520 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:49.765199 14544 log.cc:927] Time spent T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.213s user 0.002s sys 0.000s
I20260321 02:59:49.769626 14293 leader_election.cc:290] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 11 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:49.744479 14437 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.773229 14437 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:49.744305 14207 raft_consensus.cc:515] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.785449 14466 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:49.786060 14207 leader_election.cc:290] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:49.737795 14402 consensus_queue.cc:1048] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.060s
W20260321 02:59:49.797878 14558 log.cc:927] Time spent T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.143s user 0.001s sys 0.002s
I20260321 02:59:49.742292 14507 raft_consensus.cc:515] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.788213 14281 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:49.809628 14390 tablet_copy_client.cc:806] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 0 data blocks...
W20260321 02:59:49.792433 14551 log.cc:927] Time spent T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da: Append to log took a long time: real 0.342s user 0.002s sys 0.000s
I20260321 02:59:49.796741 14411 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:49.821833 14411 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
W20260321 02:59:49.830749 14537 log.cc:927] Time spent T ee109699b22b4cd08d4e2ac0ba6f3239 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.736s user 0.003s sys 0.000s
I20260321 02:59:49.832393 14507 leader_election.cc:290] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:49.883327 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 521967us
I20260321 02:59:49.887527 11200 leader_election.cc:304] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:49.893239 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
W20260321 02:59:49.895232 11232 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59202 (request call id 267) took 1480 ms (client timeout 1800 ms). Trace:
I20260321 02:59:49.895390 11231 raft_consensus.cc:1275] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260321 02:59:49.896281 11233 raft_consensus.cc:1240] T ff63bfd190b741c98793ff65c77265da P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9. Ops: []
W20260321 02:59:49.896025 11232 rpcz_store.cc:269] 0321 02:59:48.415044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:48.417375 (+ 2331us) service_pool.cc:225] Handling call
0321 02:59:49.894608 (+1477233us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b54002891d4. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8ac4b5951 00007fa8ac5506da 00007fa8b150e1d7 00007fa8aaa8f834 00007fa8aaa8f7c2 00007fa8aaa8f6f1 00007fa8aaa8f66c 00007fa8aaa8e922 00007fa8aa8895cc 00007fa8aa888a06 00007fa8aa8883af
0321 02:59:49.894905 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1140410240}
I20260321 02:59:49.897706 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:49.897763 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:49.899115 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56f96ea607c24b239b4eb847bdf1e142" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:49.900318 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:49.732909 11381 raft_consensus.cc:2468] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:49.900451 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260321 02:59:49.901469 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:49.914762 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 201994us
I20260321 02:59:49.934208 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:49.940361 14281 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:49.940168 13723 consensus_queue.cc:1048] T 62b43ed6f6404b7aa8463e168f98232a P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260321 02:59:49.941118 11233 raft_consensus.cc:1275] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260321 02:59:49.941339 14281 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.942464 14530 raft_consensus.cc:2749] T 1b8d0e26d25e4888bc95c0956520bdd1 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:49.942665 13723 raft_consensus.cc:493] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:49.949844 11236 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:49.970062 11234 raft_consensus.cc:1275] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
W20260321 02:59:49.971184 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 79344.3us
I20260321 02:59:49.970484 11234 pending_rounds.cc:85] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb: Aborting all ops after (but not including) 2
I20260321 02:59:49.977212 11231 raft_consensus.cc:2393] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 8: Already voted for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in this term.
I20260321 02:59:49.967890 14318 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:49.981331 14497 consensus_queue.cc:1048] T f9e8870ff9214a65af6cd9f677bffab4 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:49.983227 14520 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:49.984097 14520 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.985162 14495 consensus_queue.cc:1048] T 0225614ab358492abd584946a97a42f2 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260321 02:59:49.985659 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:49.986518 11236 raft_consensus.cc:3060] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Advancing to term 11
I20260321 02:59:49.986793 14520 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:49.987178 14495 consensus_queue.cc:1048] T f6a7450895514f1dad480d2716c732d9 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:59:49.988047 14402 raft_consensus.cc:476] T d42ec1d798814a50bb535a7012a76eb4 P 83b12b883a7f4129a87dadf5a78f08da [term 5 LEADER]: Not starting pre-election -- already a leader
I20260321 02:59:49.992266 14405 consensus_queue.cc:1048] T 8ac864becf784ae392582f636a51863e P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260321 02:59:49.993834 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 190862us
I20260321 02:59:49.996879 14517 consensus_queue.cc:1059] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260321 02:59:49.998257 14517 raft_consensus.cc:3055] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 11 LEADER]: Stepping down as leader of term 11
I20260321 02:59:49.998723 14517 raft_consensus.cc:740] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:49.992340 14451 consensus_queue.cc:1048] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:59:50.005812 14311 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:50.006762 14311 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.008271 14530 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:50.009091 14530 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:49.995218 14405 consensus_queue.cc:1059] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:50.014242 14405 raft_consensus.cc:3055] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Stepping down as leader of term 8
I20260321 02:59:50.015357 14405 raft_consensus.cc:740] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:50.020601 14517 consensus_queue.cc:260] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.021195 14318 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:49.977631 14309 consensus_queue.cc:1048] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
I20260321 02:59:49.974954 11234 pending_rounds.cc:107] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260321 02:59:49.826296 14390 tablet_copy_client.cc:670] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P 3d8d3176d6ec407ea2a965556c2e2c56: tablet copy: Starting download of 1 WAL segments...
I20260321 02:59:49.973004 14261 raft_consensus.cc:493] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:50.038018 11234 raft_consensus.cc:2889] T ce2a3b4534454095902245efe52aeb14 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260321 02:59:49.983521 14497 consensus_queue.cc:1059] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 2, Time since last communication: 0.065s
I20260321 02:59:50.054615 11227 raft_consensus.cc:2468] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 10.
I20260321 02:59:49.991853 14342 raft_consensus.cc:493] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:50.053340 11275 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:50.062584 14261 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.066591 14318 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.068609 11200 leader_election.cc:304] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:50.070796 14580 raft_consensus.cc:2804] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Leader election won for term 10
I20260321 02:59:50.066720 14497 raft_consensus.cc:3055] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 LEADER]: Stepping down as leader of term 10
I20260321 02:59:50.072212 14497 raft_consensus.cc:740] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:50.072381 14582 consensus_queue.cc:1059] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260321 02:59:50.060957 14281 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:50.074159 14582 raft_consensus.cc:3055] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Stepping down as leader of term 9
I20260321 02:59:50.073835 14497 consensus_queue.cc:260] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.074616 14582 raft_consensus.cc:740] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:50.076246 14582 consensus_queue.cc:260] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:50.077816 14582 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:50.080545 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:50.084429 14405 consensus_queue.cc:260] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.086292 14405 raft_consensus.cc:3060] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:50.077912 14311 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.071635 14580 raft_consensus.cc:697] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:50.096321 14411 raft_consensus.cc:515] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.097211 14580 consensus_queue.cc:237] T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.098924 14411 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.105726 14342 raft_consensus.cc:515] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:50.117611 14261 leader_election.cc:290] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.055886 14530 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.075524 14497 raft_consensus.cc:3060] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Advancing to term 11
I20260321 02:59:50.077747 14517 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 11 FOLLOWER]: Advancing to term 12
I20260321 02:59:50.137290 11227 raft_consensus.cc:2468] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 8.
I20260321 02:59:50.139004 11236 raft_consensus.cc:2468] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 11.
W20260321 02:59:50.150544 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:50.152097 14281 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:50.152330 14342 leader_election.cc:290] T 006e9dd81c0c4368bed549c97128f4ea P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:50.153002 14281 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.047534 13723 raft_consensus.cc:515] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:50.157644 14416 raft_consensus.cc:493] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:50.165256 14416 raft_consensus.cc:515] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:50.160279 13865 raft_consensus.cc:2749] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260321 02:59:50.168979 14416 leader_election.cc:290] T d42ec1d798814a50bb535a7012a76eb4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:50.161222 14507 raft_consensus.cc:2804] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260321 02:59:50.173667 14507 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 11 FOLLOWER]: Starting leader election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:50.174140 14507 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 11 FOLLOWER]: Advancing to term 12
W20260321 02:59:50.162230 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 122680us
I20260321 02:59:50.169328 14545 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:50.184120 14545 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.185473 13865 consensus_queue.cc:1048] T 36e68ffcc96a423c8e07ac7cbfaa149a P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260321 02:59:50.193513 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:50.197634 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:50.160542 14281 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:50.259271 11275 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56
I20260321 02:59:50.269754 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:50.276875 14437 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:50.278268 13865 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 8
I20260321 02:59:50.277907 14437 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.264705 13723 leader_election.cc:290] T a42977f759be4144a5862197a77e395d P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:50.290746 14533 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:50.291635 14533 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.294288 14533 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:50.281911 14545 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:50.306963 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 58841.3us
I20260321 02:59:50.327651 11385 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:50.311324 14530 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:50.331730 14298 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:50.332468 14298 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:50.332201 14530 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:50.336732 11275 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters: fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:50.339314 14309 raft_consensus.cc:2804] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Leader election won for term 10
I20260321 02:59:50.313421 14437 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:50.349073 11385 raft_consensus.cc:2393] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 11: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:50.341466 14309 raft_consensus.cc:697] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [term 10 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:50.357733 14495 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:50.363400 14495 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.363885 14530 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:50.366559 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:50.367362 11977 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
W20260321 02:59:50.365480 14591 log.cc:927] Time spent T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.068s user 0.000s sys 0.001s
I20260321 02:59:50.365568 14309 consensus_queue.cc:237] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.369035 11276 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:50.370302 14530 raft_consensus.cc:2804] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:50.370801 14530 raft_consensus.cc:493] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:50.371157 14592 log.cc:927] Time spent T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.071s user 0.001s sys 0.000s
I20260321 02:59:50.371206 14530 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:50.389581 14298 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.391010 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:50.392094 11384 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 8.
W20260321 02:59:50.384605 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 48000.2us
I20260321 02:59:50.396574 11942 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:50.397709 11275 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:50.397765 14293 consensus_queue.cc:1048] T ce2a3b4534454095902245efe52aeb14 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260321 02:59:50.399199 14407 raft_consensus.cc:2804] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 11 FOLLOWER]: Leader election won for term 11
I20260321 02:59:50.399806 14407 raft_consensus.cc:697] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [term 11 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:50.400874 14407 consensus_queue.cc:237] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.408298 14298 raft_consensus.cc:2804] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:50.412464 14298 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:50.413162 14298 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:50.413786 14507 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.421271 14507 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 12 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.437767 14495 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.440899 13865 raft_consensus.cc:2749] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260321 02:59:50.428206 14594 log.cc:927] Time spent T 4be17edc893b4dea9368c0e67b931ef6 P 0d7efc590a1041a9bd4cd4a971332dfb: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20260321 02:59:50.442612 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:50.443751 11232 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 9.
I20260321 02:59:50.443989 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3c50eb057ef548839834ecb7dfd20ce5" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:50.432269 14530 raft_consensus.cc:515] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
W20260321 02:59:50.445916 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:50.446460 14530 leader_election.cc:290] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:50.446802 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f5e054aa3c74a3282b3340fd3b1ea21" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:50.447556 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:50.447594 11229 raft_consensus.cc:2468] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 9.
I20260321 02:59:50.448144 11977 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:50.450242 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:50.451282 11229 raft_consensus.cc:2410] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260321 02:59:50.454214 11133 catalog_manager.cc:5671] T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 2 to 5, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 5 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:59:50.456477 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:50.459020 11977 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:50.460399 11276 leader_election.cc:304] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:50.465153 14309 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:50.465790 14309 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.466951 11133 catalog_manager.cc:5671] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 reported cstate change: term changed from 4 to 7, leader changed from 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1) to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 7 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } }
I20260321 02:59:50.468561 13865 raft_consensus.cc:2804] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election won for term 9
I20260321 02:59:50.471172 13865 raft_consensus.cc:697] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [term 9 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:50.471700 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:50.472234 13865 consensus_queue.cc:237] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:50.474738 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2d71c397418a46d085c9974230b7f964" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: true dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:50.475503 11232 raft_consensus.cc:2393] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 10: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:50.475309 11229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:50.479419 14495 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:50.480386 14309 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.480226 14495 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.483198 14495 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.485364 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a38bdd851a24a9a9c5e835ac457cda2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:50.486289 11228 raft_consensus.cc:2393] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 11: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:50.484239 11275 leader_election.cc:304] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:50.498888 14402 raft_consensus.cc:2749] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260321 02:59:50.501834 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:50.507477 11228 raft_consensus.cc:2393] T 3b4e9b11d3c54f328ebeeee4854be727 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 9: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:50.516386 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:50.525717 11275 leader_election.cc:304] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:50.527351 14281 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:50.527973 14281 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.528319 14402 raft_consensus.cc:2804] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:50.528224 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:50.529775 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:50.531756 14281 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:50.535261 14495 raft_consensus.cc:493] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:50.536173 14495 raft_consensus.cc:515] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:50.537791 14298 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:50.540339 11977 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:50.540885 14507 raft_consensus.cc:493] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:50.541673 14507 raft_consensus.cc:515] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:50.544301 14411 raft_consensus.cc:493] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fef7f8cc55b043039131bd9b2229f0a5)
I20260321 02:59:50.544945 14411 raft_consensus.cc:515] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:50.546066 14507 leader_election.cc:290] T 7d5b2547bde048c88e963f650d905f76 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:50.547361 14411 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:50.529429 14402 raft_consensus.cc:697] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [term 8 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:50.549263 14402 consensus_queue.cc:237] T 56f96ea607c24b239b4eb847bdf1e142 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.550139 14530 raft_consensus.cc:493] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:50.550766 14530 raft_consensus.cc:515] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.551277 14298 leader_election.cc:290] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 9 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.552646 14530 leader_election.cc:290] T c104d8b810534a6c918eab2f919a83dc P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.553982 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:50.555003 11976 raft_consensus.cc:2368] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9.
W20260321 02:59:50.555037 11349 leader_election.cc:336] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Timed out: RequestConsensusVote RPC to 127.9.213.1:36717 timed out after 1.598s (SENT)
I20260321 02:59:50.557471 11378 raft_consensus.cc:1275] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260321 02:59:50.557605 11232 raft_consensus.cc:1275] T c2f6dd25ab2545e59d5922d37e1ec68d P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260321 02:59:50.562974 11276 leader_election.cc:400] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Vote denied by peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285) with higher term. Message: Invalid argument: T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9.
I20260321 02:59:50.563496 11276 leader_election.cc:403] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260321 02:59:50.565599 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:50.566334 11236 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:50.572595 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:50.573199 11227 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
W20260321 02:59:50.577788 11182 auto_rebalancer.cc:274] failed to send replica move request: Illegal state: Replica 83b12b883a7f4129a87dadf5a78f08da is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 9 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } }
I20260321 02:59:50.580477 11385 raft_consensus.cc:1275] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260321 02:59:50.581570 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:50.582443 11381 raft_consensus.cc:2393] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 9: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:50.575574 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:50.583509 11233 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:50.584576 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:50.586001 11233 raft_consensus.cc:1240] T b10f30fc9cc94383afab2980647cbb2e P 0d7efc590a1041a9bd4cd4a971332dfb [term 12 FOLLOWER]: Rejecting Update request from peer 83b12b883a7f4129a87dadf5a78f08da for earlier term 11. Current term is 12. Ops: []
I20260321 02:59:50.594092 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:50.570830 14495 leader_election.cc:290] T 515945e5685c4afe9d41d5317acb434c P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:50.593509 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:50.596997 11227 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:50.589735 14407 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Advancing to term 9
I20260321 02:59:50.597435 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:50.605160 11236 raft_consensus.cc:3060] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:50.601253 11378 raft_consensus.cc:3060] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:50.609663 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:50.610574 11381 raft_consensus.cc:3060] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:50.639386 11234 raft_consensus.cc:1275] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260321 02:59:50.655710 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:50.626348 14517 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:50.660130 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:50.666009 13865 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:50.661608 14517 raft_consensus.cc:515] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.648499 14530 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:50.672278 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:50.677425 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:50.659008 14309 raft_consensus.cc:493] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:50.684144 11384 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:50.691381 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:50.697279 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:50.699084 11236 raft_consensus.cc:2468] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:50.666617 13865 raft_consensus.cc:515] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.703459 11236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:50.711709 11229 raft_consensus.cc:2468] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 11.
I20260321 02:59:50.686439 14530 raft_consensus.cc:515] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.693595 14309 raft_consensus.cc:515] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.743620 11134 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42514 (request call id 212) took 9510 ms. Trace:
I20260321 02:59:50.735055 14281 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:50.744974 13865 leader_election.cc:290] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.740886 14293 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:50.743731 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:50.750964 13865 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:50.744966 14309 leader_election.cc:290] T 96159c887ede40b68c4dc0ab508ada79 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.761340 14611 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:50.808332 14611 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.744549 11134 rpcz_store.cc:276] 0321 02:59:41.206497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:41.214256 (+ 7759us) service_pool.cc:225] Handling call
0321 02:59:50.704855 (+9490599us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:59:48.211359 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:59:48.212067 (+ 708us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:59:48.212111 (+ 44us) write_op.cc:435] Acquired schema lock
0321 02:59:48.212136 (+ 25us) tablet.cc:661] Decoding operations
0321 02:59:48.214324 (+ 2188us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:59:48.214438 (+ 114us) write_op.cc:641] Partition lock acquired for write op
0321 02:59:48.214482 (+ 44us) tablet.cc:684] Acquiring locks for 20 operations
0321 02:59:48.216621 (+ 2139us) tablet.cc:700] Row locks acquired
0321 02:59:48.216646 (+ 25us) write_op.cc:260] PREPARE: finished
0321 02:59:48.216874 (+ 228us) write_op.cc:270] Start()
0321 02:59:48.217019 (+ 145us) write_op.cc:276] Timestamp: P: 1774061988216793 usec, L: 0
0321 02:59:48.217043 (+ 24us) op_driver.cc:348] REPLICATION: starting
0321 02:59:48.218588 (+ 1545us) log.cc:844] Serialized 7789 byte log entry
0321 02:59:48.708616 (+490028us) op_driver.cc:464] REPLICATION: finished
0321 02:59:49.293391 (+584775us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b440002304c. stack: 00007fa8a9484413 00007fa8a9484310 00007fa8a8dd2574 00007fa8b2a876b4 00007fa8b2a87649 00007fa8a92f43a1 00007fa8a93c2dc5 00007fa8a93c37e2 00007fa8ac0f757b 00007fa8ac0f2e4f 00007fa8a9498fbc 00007fa8b2276b19 00007fa8a94b546b 00007fa8a94b370f 00007fa8a94b59d2 00007fa8aff8d810
0321 02:59:49.457630 (+164239us) write_op.cc:301] APPLY: starting
0321 02:59:49.457719 (+ 89us) tablet.cc:1366] starting BulkCheckPresence
0321 02:59:49.458150 (+ 431us) tablet.cc:1369] finished BulkCheckPresence
0321 02:59:49.458179 (+ 29us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:59:49.606449 (+148270us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:59:49.612720 (+ 6271us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=20
0321 02:59:49.618407 (+ 5687us) write_op.cc:312] APPLY: finished
0321 02:59:49.674273 (+ 55866us) log.cc:844] Serialized 139 byte log entry
0321 02:59:49.817737 (+143464us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:49.822016 (+ 4279us) write_op.cc:454] Released schema lock
0321 02:59:49.824377 (+ 2361us) write_op.cc:341] FINISH: Updating metrics
0321 02:59:49.824934 (+ 557us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:49.825005 (+ 71us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":298,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":744562,"apply.run_cpu_time_us":96169,"apply.run_wall_time_us":367202,"num_ops":20,"prepare.queue_time_us":205447,"prepare.run_cpu_time_us":7827,"prepare.run_wall_time_us":40872,"raft.queue_time_us":248657,"raft.run_cpu_time_us":23365,"raft.run_wall_time_us":768885,"replication_time_us":472469,"spinlock_wait_cycles":29949184,"thread_start_us":1125348,"threads_started":5,"wal-append.queue_time_us":162592,"wal-append.run_cpu_time_us":34052,"wal-append.run_wall_time_us":1345460}]]}
I20260321 02:59:50.767268 14517 leader_election.cc:290] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.769096 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76097e48b77445aaa1ffe490b8bb41cf" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:50.769927 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:50.811623 11234 raft_consensus.cc:2376] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 8. Re-sending same reply.
I20260321 02:59:50.754709 14437 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:50.758042 14530 leader_election.cc:290] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.788823 14545 raft_consensus.cc:493] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:50.812660 14437 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.812938 14545 raft_consensus.cc:515] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.787654 11378 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 9.
I20260321 02:59:50.814792 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "006e9dd81c0c4368bed549c97128f4ea" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:50.814990 14437 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:50.815084 14545 leader_election.cc:290] T c2f6dd25ab2545e59d5922d37e1ec68d P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:50.782102 14293 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.815922 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:50.817167 11378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:50.817816 14293 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.748256 14281 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.819777 14281 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:50.803778 11384 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:50.804944 11383 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:50.805783 11380 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:50.807410 11379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:50.810575 14611 leader_election.cc:290] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.821929 11232 raft_consensus.cc:2368] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 3d8d3176d6ec407ea2a965556c2e2c56 for earlier term 8. Current term is 9.
I20260321 02:59:50.822641 11380 raft_consensus.cc:2410] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260321 02:59:50.825516 13865 raft_consensus.cc:515] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.828330 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:50.845604 14407 raft_consensus.cc:2749] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285) with higher term. Message: Invalid argument: T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da for earlier term 8. Current term is 9.
I20260321 02:59:50.855473 11942 leader_election.cc:304] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:50.881481 13865 leader_election.cc:290] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.882196 13865 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:50.882797 13865 raft_consensus.cc:515] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.844730 14611 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:50.885732 14611 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.891362 11381 raft_consensus.cc:2468] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:50.897634 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:50.899618 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
I20260321 02:59:50.863895 14506 raft_consensus.cc:2804] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 9 FOLLOWER]: Leader election won for term 9
I20260321 02:59:50.903672 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "515945e5685c4afe9d41d5317acb434c" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:50.887743 14611 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.856835 14545 raft_consensus.cc:493] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:50.910286 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 208570us
W20260321 02:59:50.928709 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:50.902333 14506 raft_consensus.cc:697] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 9 LEADER]: Becoming Leader. State: Replica: 3d8d3176d6ec407ea2a965556c2e2c56, State: Running, Role: LEADER
I20260321 02:59:50.947203 11381 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d5b2547bde048c88e963f650d905f76" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:50.946780 14402 raft_consensus.cc:493] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:50.923081 14545 raft_consensus.cc:515] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } attrs { replace: true } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: NON_VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: true } }
I20260321 02:59:50.962598 13865 leader_election.cc:290] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.962378 14402 raft_consensus.cc:515] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.968523 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:50.964576 14402 leader_election.cc:290] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:50.946054 14437 raft_consensus.cc:493] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:50.976343 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b968fa0c5fb84474b7445627011bf12a" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:50.979449 11311 raft_consensus.cc:2468] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 9.
I20260321 02:59:50.968744 14545 leader_election.cc:290] T 8f4ec8d11b7b4e6096d8c2b8ece078dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:50.985272 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:50.988792 11349 leader_election.cc:304] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:50.990311 14281 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:50.990816 11349 leader_election.cc:304] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:50.990940 14281 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:50.993846 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:50.994843 11309 raft_consensus.cc:2468] T 6f453202f8fb47abaa1cc6c0fc390553 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 8.
I20260321 02:59:50.995276 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:50.995909 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:50.997185 11349 leader_election.cc:304] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:50.996949 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:51.004160 11304 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:51.004603 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:51.007014 11311 raft_consensus.cc:2468] T 446eeb6a12c64deb848b6db18a6587f6 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 8.
I20260321 02:59:50.999874 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d42ec1d798814a50bb535a7012a76eb4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:51.010941 11382 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "fef7f8cc55b043039131bd9b2229f0a5" is_pre_election: true
I20260321 02:59:50.997690 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3b4e9b11d3c54f328ebeeee4854be727" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:51.000784 14342 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:50.996098 14545 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:50.998230 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce2a3b4534454095902245efe52aeb14" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:51.030194 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:51.034360 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "562af69310cc422e8b378a98d1bb279b" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:50.982959 14437 raft_consensus.cc:515] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:51.037711 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:51.038002 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a42977f759be4144a5862197a77e395d" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:51.038998 11308 raft_consensus.cc:2410] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260321 02:59:51.039388 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:51.039630 14342 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:51.040066 11306 raft_consensus.cc:2468] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 9.
I20260321 02:59:51.040470 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7c6a0a98407b42d18543015ce289b654" candidate_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da"
I20260321 02:59:51.040931 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:51.041531 11306 raft_consensus.cc:2468] T f6a7450895514f1dad480d2716c732d9 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 9.
I20260321 02:59:51.041831 14342 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:51.042377 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:51.043098 11306 raft_consensus.cc:2468] T 48b15e4599b848c8a833e37849eabbda P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 9.
I20260321 02:59:51.024046 14506 consensus_queue.cc:237] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:51.048077 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:51.048982 11310 raft_consensus.cc:2393] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 9: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
I20260321 02:59:51.053045 14281 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:51.053823 14281 raft_consensus.cc:2804] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader election won for term 8
I20260321 02:59:51.054181 11972 raft_consensus.cc:1275] T 76097e48b77445aaa1ffe490b8bb41cf P 3d8d3176d6ec407ea2a965556c2e2c56 [term 9 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260321 02:59:51.054399 14281 raft_consensus.cc:697] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [term 8 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:51.055910 11306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:51.058689 11306 raft_consensus.cc:1275] T ee109699b22b4cd08d4e2ac0ba6f3239 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260321 02:59:51.074095 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6a7450895514f1dad480d2716c732d9" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:51.060715 14411 consensus_queue.cc:1048] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260321 02:59:51.039834 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36e68ffcc96a423c8e07ac7cbfaa149a" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:51.031812 14545 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:51.064965 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd5fbcf5c6ce4a819d81ed0d22bf7636" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:51.065488 14437 leader_election.cc:290] T 5aaa96ba673046439e87b58b0e693526 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:51.068141 11302 raft_consensus.cc:1275] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Refusing update from remote peer fef7f8cc55b043039131bd9b2229f0a5: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260321 02:59:51.119866 14437 raft_consensus.cc:2804] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260321 02:59:51.069584 11972 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" is_pre_election: true
I20260321 02:59:51.071162 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c104d8b810534a6c918eab2f919a83dc" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:51.089038 11304 raft_consensus.cc:2468] T 7c6a0a98407b42d18543015ce289b654 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8d3176d6ec407ea2a965556c2e2c56 in term 9.
I20260321 02:59:51.093042 11303 raft_consensus.cc:2468] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
I20260321 02:59:51.107056 11305 raft_consensus.cc:2468] T 3b4e9b11d3c54f328ebeeee4854be727 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 9.
I20260321 02:59:51.113739 11308 raft_consensus.cc:1275] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260321 02:59:51.115422 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96159c887ede40b68c4dc0ab508ada79" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:51.122187 11307 raft_consensus.cc:2393] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d7efc590a1041a9bd4cd4a971332dfb in current term 9: Already voted for candidate 83b12b883a7f4129a87dadf5a78f08da in this term.
W20260321 02:59:51.117105 14616 log.cc:927] Time spent T ce2a3b4534454095902245efe52aeb14 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20260321 02:59:51.068480 14407 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:51.126827 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48b15e4599b848c8a833e37849eabbda" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:51.132027 11972 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 8.
I20260321 02:59:51.133687 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f453202f8fb47abaa1cc6c0fc390553" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:51.055518 14281 consensus_queue.cc:237] T 467887dd1b5b410390d81755f38f0deb P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:51.132098 11236 raft_consensus.cc:1275] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260321 02:59:51.312160 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c2f6dd25ab2545e59d5922d37e1ec68d" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:51.163120 11227 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "467887dd1b5b410390d81755f38f0deb" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:51.330807 11227 raft_consensus.cc:2393] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 8: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:51.312160 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:51.155138 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "42855a31af244db9b41f42c30304de34" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:51.322103 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:51.120321 14437 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:51.335280 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 338170us
I20260321 02:59:51.191489 11201 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:51.331753 11234 raft_consensus.cc:2468] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 8.
I20260321 02:59:51.335572 11235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee109699b22b4cd08d4e2ac0ba6f3239" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:51.262530 14619 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:51.341312 11310 raft_consensus.cc:2410] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260321 02:59:51.167616 14293 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:51.191082 14545 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:51.328123 14614 raft_consensus.cc:493] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:51.341661 11201 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 178164us
I20260321 02:59:51.176589 14618 raft_consensus.cc:493] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:51.338474 14437 raft_consensus.cc:3060] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:51.401317 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:51.404035 14619 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:51.406419 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "446eeb6a12c64deb848b6db18a6587f6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:51.407717 11378 raft_consensus.cc:1275] T 4be17edc893b4dea9368c0e67b931ef6 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Refusing update from remote peer 0d7efc590a1041a9bd4cd4a971332dfb: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260321 02:59:51.408787 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fb9aee1ed8d94adbbef4eb7ac83735f1" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:51.409754 11233 raft_consensus.cc:2393] T fb9aee1ed8d94adbbef4eb7ac83735f1 P 0d7efc590a1041a9bd4cd4a971332dfb [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 83b12b883a7f4129a87dadf5a78f08da in current term 11: Already voted for candidate fef7f8cc55b043039131bd9b2229f0a5 in this term.
I20260321 02:59:51.406476 14619 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:51.396643 14618 raft_consensus.cc:515] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:51.397725 11200 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb; no voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:51.347856 14342 raft_consensus.cc:2804] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:51.426637 11232 raft_consensus.cc:1275] T 76097e48b77445aaa1ffe490b8bb41cf P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260321 02:59:51.155512 14407 raft_consensus.cc:515] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:51.442435 11349 leader_election.cc:304] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:51.413867 14619 raft_consensus.cc:493] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:51.434816 14342 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:51.446449 14411 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260321 02:59:51.475279 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 77674.1us
I20260321 02:59:51.446132 13865 raft_consensus.cc:493] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:51.428385 14545 raft_consensus.cc:493] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:51.466845 14507 raft_consensus.cc:2749] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260321 02:59:51.421825 14614 raft_consensus.cc:515] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:51.474970 14342 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:51.469405 14293 raft_consensus.cc:515] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:51.471697 14619 raft_consensus.cc:515] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:51.537945 14634 log.cc:927] Time spent T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
I20260321 02:59:51.538482 14620 raft_consensus.cc:2749] T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260321 02:59:51.533998 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 74551.5us
I20260321 02:59:51.504603 14411 raft_consensus.cc:515] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:51.553090 11275 leader_election.cc:304] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:51.544883 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 135412us
W20260321 02:59:51.556735 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 92809.1us
I20260321 02:59:51.506670 13865 raft_consensus.cc:515] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:51.514137 14545 raft_consensus.cc:515] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:51.526836 14611 consensus_queue.cc:1048] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.035s
I20260321 02:59:51.557796 14405 raft_consensus.cc:2804] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:51.587540 14638 raft_consensus.cc:493] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:51.594081 14638 raft_consensus.cc:515] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:51.594735 14635 log.cc:927] Time spent T 86406dbef163402197aa42a22bcc0101 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20260321 02:59:51.598384 14545 leader_election.cc:290] T 8ac864becf784ae392582f636a51863e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:51.555630 14507 consensus_queue.cc:1048] T 8dad9ff92f18442fa92c79c7c804610b P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260321 02:59:51.585220 14405 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:51.616895 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 46624us
I20260321 02:59:51.573098 14517 consensus_queue.cc:1059] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.033s
I20260321 02:59:51.602021 14636 raft_consensus.cc:2749] T 2d71c397418a46d085c9974230b7f964 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260321 02:59:51.624419 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 228223us
I20260321 02:59:51.624039 14405 raft_consensus.cc:3060] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:51.606683 14618 leader_election.cc:290] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:51.575443 14511 consensus_queue.cc:1059] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20260321 02:59:51.613778 14495 raft_consensus.cc:493] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:51.616510 14407 leader_election.cc:290] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:51.652415 11349 leader_election.cc:304] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:51.668720 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 93066.7us
W20260321 02:59:51.669307 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 112889us
I20260321 02:59:51.674324 11275 leader_election.cc:304] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:51.643517 14618 raft_consensus.cc:3055] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 11 LEADER]: Stepping down as leader of term 11
I20260321 02:59:51.643653 14636 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:51.664701 14644 raft_consensus.cc:493] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:51.697086 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 54808.6us
W20260321 02:59:51.699469 14633 log.cc:927] Time spent T 62b43ed6f6404b7aa8463e168f98232a P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.198s user 0.000s sys 0.003s
I20260321 02:59:51.700230 14619 leader_election.cc:290] T b10f30fc9cc94383afab2980647cbb2e P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:51.695192 14638 leader_election.cc:290] T ff63bfd190b741c98793ff65c77265da P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:51.688588 14407 raft_consensus.cc:2804] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader election won for term 9
W20260321 02:59:51.688402 14622 log.cc:927] Time spent T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.163s user 0.000s sys 0.002s
W20260321 02:59:51.704293 14630 log.cc:927] Time spent T f9e8870ff9214a65af6cd9f677bffab4 P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.158s user 0.000s sys 0.001s
I20260321 02:59:51.680965 14642 raft_consensus.cc:493] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:51.645260 14495 raft_consensus.cc:515] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:51.682668 14618 raft_consensus.cc:740] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
W20260321 02:59:51.722004 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 47194.1us
I20260321 02:59:51.725440 14645 raft_consensus.cc:2749] T 8a38bdd851a24a9a9c5e835ac457cda2 P fef7f8cc55b043039131bd9b2229f0a5 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260321 02:59:51.708760 14293 leader_election.cc:290] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:51.710168 14614 leader_election.cc:290] T 4be17edc893b4dea9368c0e67b931ef6 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
W20260321 02:59:51.716413 14631 log.cc:927] Time spent T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.050s user 0.000s sys 0.001s
I20260321 02:59:51.720404 13865 leader_election.cc:290] T 0225614ab358492abd584946a97a42f2 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:51.724655 14411 leader_election.cc:290] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:51.689875 14636 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:51.761163 11349 leader_election.cc:304] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, fef7f8cc55b043039131bd9b2229f0a5; no voters: 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:51.761345 14611 consensus_queue.cc:1048] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260321 02:59:51.768653 11275 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:51.714542 14629 consensus_queue.cc:1048] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260321 02:59:51.712780 14644 raft_consensus.cc:515] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:51.784247 14618 consensus_queue.cc:260] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:51.789844 14437 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:51.791658 14342 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:51.795291 14645 raft_consensus.cc:2804] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 11 FOLLOWER]: Leader election won for term 11
I20260321 02:59:51.797096 14636 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:51.723591 14642 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:51.799620 14495 leader_election.cc:290] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:51.800293 14580 raft_consensus.cc:3055] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Stepping down as leader of term 7
I20260321 02:59:51.809878 14402 raft_consensus.cc:2804] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260321 02:59:51.803653 14631 log.cc:927] Time spent T e626ec524e8c40ec8062293d8268e85f P fef7f8cc55b043039131bd9b2229f0a5: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
W20260321 02:59:51.825453 11202 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:51.809940 14645 raft_consensus.cc:697] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [term 11 LEADER]: Becoming Leader. State: Replica: fef7f8cc55b043039131bd9b2229f0a5, State: Running, Role: LEADER
I20260321 02:59:51.833344 14407 raft_consensus.cc:697] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [term 9 LEADER]: Becoming Leader. State: Replica: 83b12b883a7f4129a87dadf5a78f08da, State: Running, Role: LEADER
I20260321 02:59:51.837220 14342 leader_election.cc:290] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Requested vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:51.837868 14407 consensus_queue.cc:237] T fd5fbcf5c6ce4a819d81ed0d22bf7636 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:51.840620 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 108577us
I20260321 02:59:51.840760 14637 raft_consensus.cc:493] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
W20260321 02:59:51.850456 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:51.824975 14402 raft_consensus.cc:493] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:51.814455 14580 raft_consensus.cc:740] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:51.824636 14636 raft_consensus.cc:493] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:51.826542 14618 raft_consensus.cc:3060] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 11 FOLLOWER]: Advancing to term 12
I20260321 02:59:51.872527 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd04df7a33b14699ab2d2bd6325942e4" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"
I20260321 02:59:51.874153 11976 raft_consensus.cc:3060] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:51.829169 14495 consensus_queue.cc:1059] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.044s
W20260321 02:59:51.876549 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 124260us
W20260321 02:59:51.875936 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 94621.1us
I20260321 02:59:51.868592 14402 raft_consensus.cc:3060] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:51.861145 10068 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260321 02:59:51.887533 11276 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.4:41285, user_credentials={real_user=slave}} blocked reactor thread for 60873.9us
I20260321 02:59:51.892875 11275 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:51.863701 14309 consensus_queue.cc:1048] T 76097e48b77445aaa1ffe490b8bb41cf P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20260321 02:59:51.901455 11376 raft_consensus.cc:1275] T 7c6a0a98407b42d18543015ce289b654 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Refusing update from remote peer 3d8d3176d6ec407ea2a965556c2e2c56: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260321 02:59:51.872464 14636 raft_consensus.cc:515] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:51.904762 11349 leader_election.cc:304] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:51.916397 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4be17edc893b4dea9368c0e67b931ef6" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:51.918165 11232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:51.862047 14637 raft_consensus.cc:515] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:51.925899 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:51.926471 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513757b127054285b0686cd45044c596" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:51.927197 11233 raft_consensus.cc:2468] T 513757b127054285b0686cd45044c596 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 10.
I20260321 02:59:51.928421 11236 raft_consensus.cc:1275] T 8a38bdd851a24a9a9c5e835ac457cda2 P 0d7efc590a1041a9bd4cd4a971332dfb [term 11 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260321 02:59:51.929868 11132 catalog_manager.cc:5671] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:59:51.932839 14642 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:51.924525 14411 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260321 02:59:51.934132 14411 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:51.934520 14411 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:51.937613 11132 catalog_manager.cc:5671] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "83b12b883a7f4129a87dadf5a78f08da" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:59:51.939082 11132 catalog_manager.cc:5671] T e4337508daeb4d598c4f6a0b4314878d P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 5 to 8, leader changed from 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2) to 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1). New cstate: current_term: 8 leader_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:59:51.906710 14620 raft_consensus.cc:493] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:51.940618 14620 raft_consensus.cc:515] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:51.941030 11132 catalog_manager.cc:5671] T aab7e035f7984ed8ba6560e1591ec831 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:59:51.941514 14405 raft_consensus.cc:515] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } attrs { replace: false } }
I20260321 02:59:51.942842 14620 leader_election.cc:290] T 56f96ea607c24b239b4eb847bdf1e142 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:51.942535 11132 catalog_manager.cc:5671] T b270af8b9da943d2b91acbdc0bd603cc P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 6 to 7, leader changed from 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4) to fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3). New cstate: current_term: 7 leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } } }
I20260321 02:59:51.927201 11234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:51.943491 14405 leader_election.cc:290] T 404c3f80af814c3fa8db2501818b1e33 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:51.896862 14293 consensus_queue.cc:1048] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.026s
I20260321 02:59:51.955335 11228 raft_consensus.cc:1275] T 2d71c397418a46d085c9974230b7f964 P 0d7efc590a1041a9bd4cd4a971332dfb [term 10 FOLLOWER]: Refusing update from remote peer 83b12b883a7f4129a87dadf5a78f08da: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260321 02:59:51.944335 11234 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P 0d7efc590a1041a9bd4cd4a971332dfb [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 12.
I20260321 02:59:51.944970 14437 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:51.945395 14644 leader_election.cc:290] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:51.946342 14658 raft_consensus.cc:493] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:51.946842 14511 raft_consensus.cc:3055] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Stepping down as leader of term 8
I20260321 02:59:51.952193 11976 raft_consensus.cc:2468] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d7efc590a1041a9bd4cd4a971332dfb in term 9.
I20260321 02:59:51.948385 11230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0225614ab358492abd584946a97a42f2" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
W20260321 02:59:51.954301 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 46337.3us
I20260321 02:59:51.947806 14637 leader_election.cc:290] T 36e68ffcc96a423c8e07ac7cbfaa149a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:51.959479 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 54794.2us
W20260321 02:59:51.961441 11133 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42530 (request call id 220) took 17778 ms (client timeout 15000 ms). Trace:
I20260321 02:59:51.898442 14611 consensus_queue.cc:1048] T c2f6dd25ab2545e59d5922d37e1ec68d P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
I20260321 02:59:51.963019 14636 leader_election.cc:290] T 562af69310cc422e8b378a98d1bb279b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:51.956993 11132 catalog_manager.cc:5671] T 72e86c671bb747c785d8f676ed3b8315 P 3d8d3176d6ec407ea2a965556c2e2c56 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260321 02:59:51.969659 14643 raft_consensus.cc:493] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 3d8d3176d6ec407ea2a965556c2e2c56)
I20260321 02:59:51.974779 11349 leader_election.cc:304] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:51.981389 11310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f4ec8d11b7b4e6096d8c2b8ece078dc" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:51.982375 11349 leader_election.cc:304] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:51.983220 11303 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8dad9ff92f18442fa92c79c7c804610b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:51.923513 14506 consensus_queue.cc:1048] T 7c6a0a98407b42d18543015ce289b654 P 3d8d3176d6ec407ea2a965556c2e2c56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260321 02:59:51.983790 11303 raft_consensus.cc:2468] T 8dad9ff92f18442fa92c79c7c804610b P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 8.
I20260321 02:59:51.984321 11311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca80e3e3473540618f654536751cdc3b" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:51.958330 14642 raft_consensus.cc:2804] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260321 02:59:51.990043 11311 raft_consensus.cc:2468] T ca80e3e3473540618f654536751cdc3b P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 9.
W20260321 02:59:52.006124 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 55170.3us
I20260321 02:59:52.007722 11305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ac864becf784ae392582f636a51863e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:52.010026 11304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b10f30fc9cc94383afab2980647cbb2e" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:51.970191 14511 raft_consensus.cc:740] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:51.901943 14645 consensus_queue.cc:237] T fb9aee1ed8d94adbbef4eb7ac83735f1 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:51.923138 14580 consensus_queue.cc:260] T 56f96ea607c24b239b4eb847bdf1e142 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:52.027067 11302 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e8b6fd0841c644779c26179d18ddf697" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 8 candidate_status { last_received { term: 7 index: 3 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:52.098951 11349 leader_election.cc:304] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:52.119925 11309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5fc517dbbd646bfa0d307d4e42556ba" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:52.097846 11307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff63bfd190b741c98793ff65c77265da" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:52.121707 11307 raft_consensus.cc:2393] T ff63bfd190b741c98793ff65c77265da P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef7f8cc55b043039131bd9b2229f0a5 in current term 9: Already voted for candidate 0d7efc590a1041a9bd4cd4a971332dfb in this term.
I20260321 02:59:52.121088 11349 leader_election.cc:304] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:52.121138 11308 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5aaa96ba673046439e87b58b0e693526" candidate_uuid: "fef7f8cc55b043039131bd9b2229f0a5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "83b12b883a7f4129a87dadf5a78f08da" is_pre_election: true
I20260321 02:59:52.123487 11308 raft_consensus.cc:2468] T 5aaa96ba673046439e87b58b0e693526 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 9.
I20260321 02:59:52.121064 11309 raft_consensus.cc:2468] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 12.
W20260321 02:59:52.120234 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 143806us
W20260321 02:59:52.132867 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 125557us
I20260321 02:59:52.137413 11349 leader_election.cc:304] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
W20260321 02:59:52.140483 11942 leader_election.cc:336] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717): Timed out: RequestConsensusVote RPC to 127.9.213.1:36717 timed out after 1.800s (SENT)
I20260321 02:59:52.099089 11275 leader_election.cc:304] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:51.989758 14637 raft_consensus.cc:2804] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260321 02:59:52.148324 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 49305.2us
I20260321 02:59:52.149408 11942 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3d8d3176d6ec407ea2a965556c2e2c56; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:52.152127 11275 leader_election.cc:304] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
W20260321 02:59:51.974514 11133 rpcz_store.cc:269] 0321 02:59:34.182919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0321 02:59:34.183630 (+ 711us) service_pool.cc:225] Handling call
0321 02:59:51.961357 (+17777727us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0321 02:59:50.701130 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0321 02:59:50.745964 (+ 44834us) write_op.cc:432] Acquiring schema lock in shared mode
0321 02:59:50.746020 (+ 56us) write_op.cc:435] Acquired schema lock
0321 02:59:50.746057 (+ 37us) tablet.cc:661] Decoding operations
0321 02:59:50.746590 (+ 533us) write_op.cc:620] Acquiring the partition lock for write op
0321 02:59:50.746649 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0321 02:59:50.746683 (+ 34us) tablet.cc:684] Acquiring locks for 2 operations
0321 02:59:50.747143 (+ 460us) tablet.cc:700] Row locks acquired
0321 02:59:50.747161 (+ 18us) write_op.cc:260] PREPARE: finished
0321 02:59:50.747360 (+ 199us) write_op.cc:270] Start()
0321 02:59:50.747514 (+ 154us) write_op.cc:276] Timestamp: P: 1774061990747276 usec, L: 0
0321 02:59:50.747538 (+ 24us) op_driver.cc:348] REPLICATION: starting
0321 02:59:50.812010 (+ 64472us) log.cc:844] Serialized 917 byte log entry
0321 02:59:51.530469 (+718459us) op_driver.cc:464] REPLICATION: finished
0321 02:59:51.791167 (+260698us) write_op.cc:301] APPLY: starting
0321 02:59:51.791294 (+ 127us) tablet.cc:1366] starting BulkCheckPresence
0321 02:59:51.791441 (+ 147us) tablet.cc:1369] finished BulkCheckPresence
0321 02:59:51.791579 (+ 138us) tablet.cc:1371] starting ApplyRowOperation cycle
0321 02:59:51.792076 (+ 497us) tablet.cc:1382] finished ApplyRowOperation cycle
0321 02:59:51.792396 (+ 320us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0321 02:59:51.792446 (+ 50us) write_op.cc:312] APPLY: finished
0321 02:59:51.792751 (+ 305us) log.cc:844] Serialized 29 byte log entry
0321 02:59:51.792959 (+ 208us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:51.793056 (+ 97us) write_op.cc:454] Released schema lock
0321 02:59:51.793116 (+ 60us) write_op.cc:341] FINISH: Updating metrics
0321 02:59:51.793765 (+ 649us) write_op.cc:489] Releasing partition, row and schema locks
0321 02:59:51.793804 (+ 39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":216601,"reported_tablets":38,"child_traces":[["op",{"apply.queue_time_us":256595,"apply.run_cpu_time_us":2563,"apply.run_wall_time_us":2562,"num_ops":2,"prepare.queue_time_us":149450,"prepare.run_cpu_time_us":4616,"prepare.run_wall_time_us":129854,"replication_time_us":757977,"thread_start_us":175685,"threads_started":2}]]}
I20260321 02:59:52.011842 14636 raft_consensus.cc:2749] T 96159c887ede40b68c4dc0ab508ada79 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:52.005508 14642 raft_consensus.cc:493] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:52.164719 11349 leader_election.cc:304] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:52.142760 14619 raft_consensus.cc:2804] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:52.147668 14620 raft_consensus.cc:2749] T c104d8b810534a6c918eab2f919a83dc P fef7f8cc55b043039131bd9b2229f0a5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260321 02:59:52.150945 14637 raft_consensus.cc:493] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:52.177560 11349 leader_election.cc:304] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:52.150671 14663 raft_consensus.cc:493] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:52.150595 14638 raft_consensus.cc:2749] T 6e21cb03b1624c9fa5acfa1cd5d5db8f P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:52.176007 14437 raft_consensus.cc:2804] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260321 02:59:52.159539 14614 raft_consensus.cc:2804] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260321 02:59:52.182751 14614 raft_consensus.cc:493] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:52.182431 11349 leader_election.cc:304] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
I20260321 02:59:52.183189 14614 raft_consensus.cc:3060] T 513757b127054285b0686cd45044c596 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:52.186664 14646 raft_consensus.cc:2804] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260321 02:59:52.187168 14646 raft_consensus.cc:493] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:52.187584 14646 raft_consensus.cc:3060] T 48b15e4599b848c8a833e37849eabbda P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:52.182477 14437 raft_consensus.cc:493] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:52.189116 14437 raft_consensus.cc:3060] T ee109699b22b4cd08d4e2ac0ba6f3239 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:52.188956 14545 consensus_queue.cc:1048] T 42855a31af244db9b41f42c30304de34 P fef7f8cc55b043039131bd9b2229f0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260321 02:59:52.160337 14611 raft_consensus.cc:493] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:52.173739 14619 raft_consensus.cc:493] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:52.164116 14642 raft_consensus.cc:3060] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:52.163965 14517 raft_consensus.cc:2804] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260321 02:59:52.201550 11201 leader_election.cc:304] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 3d8d3176d6ec407ea2a965556c2e2c56; no voters:
I20260321 02:59:51.989820 14643 raft_consensus.cc:515] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:51.978361 14658 raft_consensus.cc:515] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:52.161893 13865 consensus_queue.cc:1048] T 8a38bdd851a24a9a9c5e835ac457cda2 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.032s
I20260321 02:59:52.180752 14637 raft_consensus.cc:3060] T 6f453202f8fb47abaa1cc6c0fc390553 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:52.181492 14309 raft_consensus.cc:493] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:52.190421 14636 raft_consensus.cc:2804] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260321 02:59:52.224268 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 97669.4us
I20260321 02:59:52.203166 14619 raft_consensus.cc:3060] T 446eeb6a12c64deb848b6db18a6587f6 P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:52.202390 14517 raft_consensus.cc:493] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260321 02:59:52.241811 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 108200us
I20260321 02:59:52.225669 14636 raft_consensus.cc:493] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
I20260321 02:59:52.375411 14636 raft_consensus.cc:3060] T f6a7450895514f1dad480d2716c732d9 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Advancing to term 10
I20260321 02:59:52.190989 14511 consensus_queue.cc:260] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:52.312510 14657 raft_consensus.cc:2749] T bd04df7a33b14699ab2d2bd6325942e4 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260321 02:59:52.224177 14663 raft_consensus.cc:515] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:52.238720 14309 raft_consensus.cc:515] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:52.223942 14611 raft_consensus.cc:515] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:52.223583 14342 raft_consensus.cc:493] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:52.223379 14649 raft_consensus.cc:493] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260321 02:59:52.192574 14654 raft_consensus.cc:493] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 0d7efc590a1041a9bd4cd4a971332dfb)
W20260321 02:59:52.399443 11275 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.1:36717, user_credentials={real_user=slave}} blocked reactor thread for 247387us
W20260321 02:59:52.400627 11277 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260321 02:59:52.413923 14611 leader_election.cc:290] T e626ec524e8c40ec8062293d8268e85f P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:52.414063 14531 raft_consensus.cc:493] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:52.385249 14517 raft_consensus.cc:3060] T c5fc517dbbd646bfa0d307d4e42556ba P 83b12b883a7f4129a87dadf5a78f08da [term 12 FOLLOWER]: Advancing to term 13
I20260321 02:59:52.415766 14309 leader_election.cc:290] T 86406dbef163402197aa42a22bcc0101 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:52.416261 14342 raft_consensus.cc:515] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
W20260321 02:59:52.404760 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 219333us
I20260321 02:59:52.438900 11233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "404c3f80af814c3fa8db2501818b1e33" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
I20260321 02:59:52.439307 11349 leader_election.cc:304] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb
W20260321 02:59:52.440955 11942 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.3:43533, user_credentials={real_user=slave}} blocked reactor thread for 209885us
I20260321 02:59:52.444624 11231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e21cb03b1624c9fa5acfa1cd5d5db8f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:52.405253 14411 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P 83b12b883a7f4129a87dadf5a78f08da [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:52.438385 14655 raft_consensus.cc:2804] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election won for term 9
W20260321 02:59:52.484405 11349 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 45245.4us
I20260321 02:59:52.485823 11349 leader_election.cc:304] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef7f8cc55b043039131bd9b2229f0a5; no voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da
I20260321 02:59:52.487490 11349 leader_election.cc:304] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83b12b883a7f4129a87dadf5a78f08da, fef7f8cc55b043039131bd9b2229f0a5; no voters:
I20260321 02:59:52.447674 14611 consensus_queue.cc:1048] T 2d71c397418a46d085c9974230b7f964 P 83b12b883a7f4129a87dadf5a78f08da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
W20260321 02:59:52.482227 14618 consensus_meta.cc:287] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.609s user 0.013s sys 0.003s
I20260321 02:59:52.425875 14649 raft_consensus.cc:515] T bd04df7a33b14699ab2d2bd6325942e4 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:52.474915 14620 raft_consensus.cc:2804] T 3b4e9b11d3c54f328ebeeee4854be727 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader election won for term 9
I20260321 02:59:52.563730 11304 raft_consensus.cc:2468] T b10f30fc9cc94383afab2980647cbb2e P 83b12b883a7f4129a87dadf5a78f08da [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fef7f8cc55b043039131bd9b2229f0a5 in term 12.
W20260321 02:59:52.562011 11200 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.9.213.2:34267, user_credentials={real_user=slave}} blocked reactor thread for 177309us
I20260321 02:59:52.463086 14642 raft_consensus.cc:515] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:52.438763 14531 raft_consensus.cc:515] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:52.532248 14643 leader_election.cc:290] T 70e1ea01442f4d13a0f6432ea463c5df P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:52.579711 11200 leader_election.cc:304] T 42855a31af244db9b41f42c30304de34 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d7efc590a1041a9bd4cd4a971332dfb, 83b12b883a7f4129a87dadf5a78f08da; no voters:
I20260321 02:59:52.537335 14655 raft_consensus.cc:697] T bd04df7a33b14699ab2d2bd6325942e4 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 LEADER]: Becoming Leader. State: Replica: 0d7efc590a1041a9bd4cd4a971332dfb, State: Running, Role: LEADER
I20260321 02:59:52.571273 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e626ec524e8c40ec8062293d8268e85f" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:52.583890 11228 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86406dbef163402197aa42a22bcc0101" candidate_uuid: "83b12b883a7f4129a87dadf5a78f08da" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" is_pre_election: true
I20260321 02:59:52.440770 14654 raft_consensus.cc:515] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:52.456573 11236 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2fcb8dea530d4005a0101183a89b709d"
dest_uuid: "0d7efc590a1041a9bd4cd4a971332dfb"
mode: GRACEFUL
new_leader_uuid: "fef7f8cc55b043039131bd9b2229f0a5"
from {username='slave'} at 127.0.0.1:41224
I20260321 02:59:52.585027 11236 raft_consensus.cc:606] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Received request to transfer leadership to TS fef7f8cc55b043039131bd9b2229f0a5
I20260321 02:59:52.585389 11236 raft_consensus.cc:614] T 2fcb8dea530d4005a0101183a89b709d P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260321 02:59:52.586382 14654 leader_election.cc:290] T f9e8870ff9214a65af6cd9f677bffab4 P 83b12b883a7f4129a87dadf5a78f08da [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:52.603092 14342 leader_election.cc:290] T 2f5e054aa3c74a3282b3340fd3b1ea21 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:52.604751 14663 leader_election.cc:290] T e8b6fd0841c644779c26179d18ddf697 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:52.606652 14531 leader_election.cc:290] T 00eeadd5c566459b86e97f920a91c398 P 3d8d3176d6ec407ea2a965556c2e2c56 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
I20260321 02:59:52.567239 14629 raft_consensus.cc:2804] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260321 02:59:52.608052 14674 raft_consensus.cc:493] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 83b12b883a7f4129a87dadf5a78f08da)
I20260321 02:59:52.608424 14629 raft_consensus.cc:493] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260321 02:59:52.608831 14629 raft_consensus.cc:3060] T 8dad9ff92f18442fa92c79c7c804610b P fef7f8cc55b043039131bd9b2229f0a5 [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:52.608640 14674 raft_consensus.cc:515] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "3d8d3176d6ec407ea2a965556c2e2c56" member_type: VOTER last_known_addr { host: "127.9.213.4" port: 41285 } attrs { promote: false } }
I20260321 02:59:52.610500 14511 raft_consensus.cc:3060] T 86406dbef163402197aa42a22bcc0101 P 0d7efc590a1041a9bd4cd4a971332dfb [term 8 FOLLOWER]: Advancing to term 9
I20260321 02:59:52.611001 14674 leader_election.cc:290] T 1146889bd4dc49e9a2b67700208bc264 P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), 3d8d3176d6ec407ea2a965556c2e2c56 (127.9.213.4:41285)
I20260321 02:59:52.613132 14642 leader_election.cc:290] T b968fa0c5fb84474b7445627011bf12a P fef7f8cc55b043039131bd9b2229f0a5 [CANDIDATE]: Term 10 election: Requested vote from peers 0d7efc590a1041a9bd4cd4a971332dfb (127.9.213.1:36717), 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267)
W20260321 02:59:52.613433 14402 consensus_meta.cc:287] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da: Time spent flushing consensus metadata: real 0.715s user 0.010s sys 0.000s
I20260321 02:59:52.562353 14658 leader_election.cc:290] T 467887dd1b5b410390d81755f38f0deb P 0d7efc590a1041a9bd4cd4a971332dfb [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 83b12b883a7f4129a87dadf5a78f08da (127.9.213.2:34267), fef7f8cc55b043039131bd9b2229f0a5 (127.9.213.3:43533)
I20260321 02:59:52.613878 14402 raft_consensus.cc:515] T 42855a31af244db9b41f42c30304de34 P 83b12b883a7f4129a87dadf5a78f08da [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" member_type: VOTER last_known_addr { host: "127.9.213.1" port: 36717 } } peers { permanent_uuid: "83b12b883a7f4129a87dadf5a78f08da" member_type: VOTER last_known_addr { host: "127.9.213.2" port: 34267 } } peers { permanent_uuid: "fef7f8cc55b043039131bd9b2229f0a5" member_type: VOTER last_known_addr { host: "127.9.213.3" port: 43533 } }
I20260321 02:59:52.620362 11233 raft_consensus.cc:2468] T 404c3f80af814c3fa8db2501818b1e33 P 0d7efc590a1041a9bd4cd4a971332dfb [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83b12b883a7f4129a87dadf5a78f08da in term 9.
I20260321 02:59:52.558207 14638 raft_consensus.cc:2749] T 0225614ab358492abd584946a97a42f2 P fef7f8cc55b043039131bd9b2229f0a5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260321 02:59:52.641412 11976 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1146889bd4dc49e9a2b67700208bc264" candidate_uuid: "0d7efc590a1041a9bd4cd4a971332dfb" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "3d8d3176d6ec407ea2a965556c2e2c56"